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Preface 



Although the dieme of the 1967 Western Regional Conference on 
Testing Problems refers to a **congeries," the papers presented 
belied that term. They turned out to be much more parts of a unified 
whole than a mere aggregation. 

The program began with Carole Leland discussing the ways in 
which measurement of the individual can be used to enhance the 
educational process with special emphasis on credit by examination 
at the college level Vincent Campbell then described assessment, 
not of individuals, but of the total educational effort as a necessary 
step in knowing what v/e are now doing so tfiat we can make 
meaningful changes if required. In all of this, the problem of the 
criterion is critical and Scarvia Anderson called attention to this and 
described approaches to better statements of criteria. Rodi>0y Skager 
then posed some interesting questions concerning the limitations and 
strengths of educational researchers in problems of evaluation. The 
final paper by Leland Newcomer provided a fitting close to tl.e 
conference by drawing attention to the role of the administrator as 
a responsible person in innovation and by pointing out the dangers 
inherent in letting what has been innovation become routine. 

Taken together, the various papers made a challenging but still 
hopeful picture of the role diat measurement and evaluation can 
play in insuring tfiat educational change becomes educational im- 
provement. 

John S. Helmick^ Chaiman 
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The Atj^cal Student: 
His Alternatives and Ours 

CAROLE A. LELAND 



Like many of you, I have spent a gcx>d deal of time in the last few 
weeks in the educators version of spring training— the myriad of 
conferences, annual meetings and hastily assembled committee ses- 
sions which constitute our warm-up for a new season. We watch our 
professional colleagues flex their muscles, so to speak, after the 
lonely, cold wintar. We observe Ae scramble for new players and 
the fanfare of illustrious signing. And we note, often with dismay, 
that the leagues are growing bigger, die fans noisier, and Ae benches 
harder. Given this exposure it is extremely difficult to visualize how 
I might avoid saying the same thin^ I have heard, thus continuing 
that endless cycle of issues and questions which seem to characterize 
our educational enterprise. (I might say too, just to stretch this 
analogy a little bit, that I am not quite sure anymore when s{»ing 
training ends and the season begins.) 

In addition to this possibility for repetition let me admit from the 
outeet, something which could b«Jome increasin^y obvious, Aat in 
all my collegiate experiences I have nevar had a course in public 
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Speaking. I have consulted with a friend who has, however, and I do 
know that one might classify speeches in four categories: argumen- 
tative, persuasive, informative and entertaining. Now, it seemed 
logical to me that if I covered all four in one speech I might com- 
pensate for my lack of formal training* 

The entertaining function I prefer to dispense with first and 
quickly. At this early hour I am sure you wouldn't expect more. Two 
stories seem appropriate to me, one which I find relevant to students 
and one which seems to have some relevance to the learning process. 
Tlie first concerns two cwvs grazing in the meadow when a dairy 
tiiick passes nearby. Tlic side of the truck announces grandly, 
"Grade A .Milk, fortified with Vitamin D, B, and B^, enriched, 
homogcni'/ed, pasteurized, lower in fat, higher in protein than regu- 
lar milk.'' One cow turned to the odier and said, "Sort of makes you 
feel inadequate, doesn't it?" 

The second story concerns a woman who took and successfully 
completed the Red' Cross's course in first aid. Some^monthr later she 
was moved to waite the national director of the Red Cross in Wash- 
ington. Her letter went like this: 

Dear Sir: 

I am exU-cmely grateful for having had tliis opportunity to take the 
Red Cross course in first aid and I want to tell you how meaningful 
it has been for me. Very recently we had a horrible accident near 
our home, with bodes strewn all over, wreckage all about and be- 
cause I had had your course in first aid I knew e.Kactly what to do. 
I put my head between my le^ and I didn't faint 

The presumptuous title which appears on your program is exactly 
what you might expect, the result of pre-planning when a speaker 
agrees to anything that is several weeks off, and the person doing 
the program accepts anything in order to get it to the press. For 
who among us to<lay can distinguish confidently between the atypi- 
cal and the typical student? Most of us arc hard put to distinguish 
between male and female, student and nonstudent, and faadty and 
administrators on the college campus* 

Nonetheless, I prefer not to abandon the topic and only wish to 
expand it a bit to read. The Atypical Student-His Alternatives 
and Ours." 

What I have in mind is to argue a common point of view about 
American Higher Education, to persuade you to delineate more 
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carefully the uniqueness of the students you deal with and in the 
process to recognize individuals who may be excluded unjustly from 
our colleges and universities, and finally, to inform you of the Col- 
lege Boards concern for providing alternatives to a diversitj' of 
individuals who seek collegiate instniction. 

Now the argument is neither original nor complicated. For all its 
magnitude and magnificence, American higher education has re- 
sponded lethargically to the challenges of an increasingly diverse 
college population. This is hardly "kicking a man when he's aown," 
for higher education has never been so "up " With larger numbers 
of students, greater interest and support from large segments of 
society and unbelievable concentrations of brain power in their 
students and faculties, the 2,000 odd colleges and universities of 
this country are, to put it mildly, ''loaded.'' To be sure, they are 
loaded with problems as well as potentials, but to rationalize the 
dormancy of the latter in the face of the former is sheer delusion. 
Admittedly, too, some institutions are making efforts to accommo- 
date all kinds of students whose motives, dispositions and capacities 
differ, sometimes markedly. We have welcomed the residential com- 
plexes of Santa Cruz or Michigan State, the technological daring of 
Oakland Community College and the overseas campuses of numer- 
ous schools as evidences of a willingness to meet the challenges of 
diversity. But, in the main, when viewed collectively, the colleges 
and universities have exhibited little passion and ingenuity in en- 
gaging large numbers of students in the learning process. As a matter 
of fact, large numbers of students have been repelled and some have 
never been attracted to the campuses at all. 

Martin Meyerson, writing in the recent volume The Contemporary 
University: USA, suggests that in our culture we tend to cater to 
majority tastes and choices in most fields, and higher education is no 
exception. He opines: 

Yet, if tlie number of students in higher education increases as rap- 
idly as we expect, in about 15 years the present number of students 
may Ihj only half the total. With such vast numbers of college and 
university students, minority interests of student subcultures could 
fill many canipu.ses. One of tlie prerequisites for high quality will be 
the provision of more educational diversity for these minority in- 
terests. 

And he concludes: ''A new academic ethos of diversity and yet 



4 CAROLE A. LELAND 



community wUl require far more spontaneity in organization than 
educational institutions have commonly exhibited 

Mr Meyerson conveys the sense of diversity but too little of the 
urgency some of us feel at the present time. Earlier I side-stepped 
the matter of "atypical " implying that our sense of the typical has 
been buried in the flutter and flurry of rapidly changing and grow- 
ing college populations. Certainly that may be true but such caution 
hardly gets us into a position of intellectual valor and challenge. 
So let me suggest that one way to view the atypical student is to 
consider him as someone who doesn't quite "fit" the traditional pat- 
tern of undergraduate education in most of our colleges and uni- 
versities. Of course, this forces us to view a good many students as 
atypical. And that's the point. 

If I am to persuade you to recognize and to cultivate the atypical 
let me select a few examples among the many available to us. (And 
let me just digress a moment here to say that probably the most dis- 
appointing part of any speech is the necessity of selecting examples 
and frequently leaving behind the one that might have been the 
most exciting and the most dramatic.) The possibilities which could 
challenge oiu: institutional sensibilizes are numerous-the drop-outs, 
the uncommitted, the alienated, the "retread** to name but a few. 
Let me select three groups, however, which are perhaps familiar 
and more common to all our ^periences but which should provoke, 
if they haven't already, both our concern and oiu: ingenuity. Each 
group, either by its own urging, or in the li^t of its unique situa- 
tion, speaks to the inadequacies of the institutions of hi^er learning. 

The student activists deserve mention, not because they haven't 
received oiu: attention (they are far too vociferous and we are far 
too curious to have missed them), but because they constitute one 
of the few groups of students we know much about. Kenneth Ken- 
niston calls attention to our vast ignorance of students in the follow- 
ing quotation which is taken again from The Contemporary Univer- 
sity: USA: 

To be sure, everyone acknowledges that v/ithout students there can 
be no university; and so too, ''education'* is widely admitted to be 
one of the functions of a university. Yet the characteristics of studente 
-the fact that tfiey have commitments, aspirations, values, dreams, 
needs, psyches and perhaps souls even before being admitted to 
college— are largely ignored in the concentration on more easfly de- 
scribable features of the university. To many administrators and to 
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some faculty members, students remain a kind of unleavened lump 
to be molded by the university, blank cards on which education will 
punch imperishable information, shapeless ingpts to be pressed into 
useful forms by "the college experience.'^ 

In contrast to this paucity of knowledge about students generally, 
we know a good deal about the student activists, particularly those 
in the 1964 FSM and those students who demonstrated at Chicago 
and Pennsylvania State. In a monograph prepared for the U.S. 
0£Sce of Fducation, Joseph Katz emphasizes that in numerous 
studies the major findings are essentially convergent And I am sure 
you will agree with all of us in research that this is amazing. Katz 
summarizes these data in the following way: The activist tends to 
come from homes with parents whose incomes, occupational status 
and education are hi^er than parents of non-activists and the par- 
ents are likely to be politically more liberal^ more permissive in 
child-rearing and to have had closer affective relationships with 
their children. In terms of academic aptitude, performances and 
attitudes, the students themselves scored significantly higher in 
verbal, not in mathematical aptitude and their grade point averages 
were significantly hi^er than those of non-activists. They scored 
higher on scales measuring theoretical orientation, liking for reflec- 
tive thought, diversity of interests and estheticism. The activists re- 
ported themselves more often than non-activists as having been 
influenced by ideas presented in courses and by teachers. Measures 
of personality characteristics and values, among other things, re- 
vealed the activists to be more flexible, tolerant and realistic, less 
dependent upon authority, rules or rituals for managing social rela- 
tionshij^y less judgmental and tending to express impulses more 
freely either in conscious thought or in overt action. In his descrip- 
tion of the activists, Katz suggests : 

That the activists are recruited particularly from the intellectually 
able and inta-ested students may well be explained by the observa- 
tion that these are students who care for the intellectual values im- 
plied in the university, who have been stimulated by the pursuit of 
truth and expression of het^odox ideas in their courses, who now 
want an extension and deepening of these experiences and who 
would like to relate ideas and theories co their own lives and to the 
improvement of society around them.^ 

We continue to observe and to eissess the impact of the student ac- 
tivists, but above all we recognize in their demands for educational 
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reforms n desire for our conceiu and commitment to them as indi- 
viduals. However an'pical their methodologies, their demonstrations 
should call attention to the rigid pathways we have constnicted 
throughout the under^aduate curriculum. 

For a second group of students, tlie collegiate experience is often 
equally fnistrating and straining, but there have been too few of 
them to constitute more than a silent minority on the campus and 
as individuals we know little about them. For the talented student 
in the arts, higher educatio!i has made only token gestures of aware- 
ness. To be sure, we see more art exhibits, more musical productions 
and dramatic events and more artists in residence on some cam- 
puses. And some colleges, Bennington and Sarah Lawrence for ex- 
ample, phce a premium on the place of arts and artists within the 
society and within the gates of the college. 

But, in a position paper prepared for the National Foundation of 
the Arts and Himianities, Lewis Mayliew dramatically presents the 
argument that while the fine and performing arts seem ascendant 
In American society, American higher education has yet to demon- 
strate tliat the arts should feature strongly in the life of each college 
and university. Such a demon-stration, lie suggests, involves both 
pro\ iding a comfortable setting for the artist within a college and 
ensuring that students with interests cr talents in the arts can enter 
the university without undue guilt or without jeopardizing their 
primary interests. Professor Mayhew bases his somewhat pessimistic 
assessment on three indications: 

L The fact that tl^e normal predictors of academic success, like high 
^school rank and performance on aptitude tests, used to screen 
students for higher education bear little or no relation.ship either 
to performance in the arts or to creativity in the arts, and interesf 
or talent in the arts is rarely among published admission criteria. 

2. That the collegiate curricula themselves reflect little concern for 
the performing arts and few college honors programs are designed 
in the arts« and 

3. The continuing posture of most collegiate faculty that the arts 
are frills in comparison with other subjects.'^ 

In like fashion. Dr. Benjamin Steigman, for 22 years principal of 
New York's High School of Music and Art, makes the point that any- 
thing like parity for high school music and arts courses is out of the 
question. He states, "The college catalogs appear to be liberal. In 
most of them there is this reassurance, 'A candidate whose record 
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shows special ability, but who lacks the required academic credit 
may present his case for consideration by the Committee on Admi. 
sions/ With four or five times as many applicants to the prestige 
colleges as they can admit, what student -^vould add to his hazards 
by lacking the required academic credits? Just how would his spe- 
cial ability ht evaluated? No, he had better stick to those courses he 
knows are acceptable/'® And, I might add that Dr. Steigman sub- 
stantiates his position with the report that out of 110 well known 
colleges, 90 percent admitted they would allow no substitution of 
music and art for either math or foreign languages. 

The student of the arts, like the student ''ctivist, is atypical in 
most of our colleges and universities because too often his "fit" to 
the U'aditional curriculum and pattern of campus life forces compro- 
nnses upon him and frequently forces him off the campus. In a 1961 
address to t' > Association of Graduate Schools, W. McNeil Lowry 
called for a * radical shift in atmosphere" surrounding students cc**- 
sidered potential artisv^. He elaborated that the essence of this at- 
moi^phere is "the artist's acceptance of concentration, even of dis- 
tortion, as a way of life, a way of life that in many ways is completely 
anti<^' Heal to the ideal objective of a liberal and humane educa- 
tioi espite some attention in individual institutions and by some 
recognized leaders, when change is measured in such terms as Mr. 
Lowry 's, our collegiate enterprise has shown relatively little concern 
for individuals in the arts or for those who might be in the arts if 
the barriers were not so great. 

Time prevents more than a passing comment or two about a third 
group of students, some of whom we recognize on our campuses, 
others we never see. Increasingly since the late 19th century the 
adult student has struggled for attention in colleges and universities 
primarily developed for educating the young. That struggle has not 
been easy, nor has it been won. The 1961-62 National Opinion Re- 
search Center survey revealed that some 25 million adults were 
engaged in some educational course or acUvitv. However, churches 
and synagogues served 3 million of these students, compared to 2.6 
million in college and university classes and 8 million were involved 
in independent study.*^ A majority of the adult students have en- 
rolled in traditionally vocational courses. 

On some campuses the adult student has been welcomed, encour- 
aged and his purposes in approaching the institution have been 
respected. Special degree programs at schools like Brooklyn, Queens, 
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Syracuse and Oklahoma have accredited previous learning and 
experience, provided flex-oility in course schedules and ensured 
the involvement of students with top-level faculty. Experiences 
with adults have proved not only satisfying but invigorating for 
many schools. In a progress report of the Center for Continuing 
Education and Community Studies at Sarah Lawrence College, 
B. J. Loewenberg reports: *The experience of the Center has justi- 
fied its initial purposes. The performance of mature women return- 
ing to undergraduate work and professional study has confirmed 
our faith in their undiminished intellectual vigor." ^ 

Swift appraisal of the adult's situation with respect to hi^er edu- 
cation suggests progress and encouragement. But more careful 
scrutiny underscores that whatever responses we have made to the 
adults who seek our attention have been minimal and sometimes 
superficial. We know very little about adults as students; we have 
been largely intolerant of their necessity for institutional flexibility 
and we seldom view the adult as an individual whose previous 
experiences have relevance or respectibility in the academic world. 
If you doubt these accusations, talk to a young housewife who has 
completed three years of college and wants to complete her educa- 
tion at a nearby university which refuses part time students, or 
listen to the militarj' man who seeks a hi^er degree and is turned 
away because he has reached his 45th birthday. 

The question really isn't who is typical and who is not. Tlie point 
is that the educational establishment is full of individuals who for 
one reason or another don't exactly fit our carefully devised four- 
year curricula. We may also do great injustice to that seemingly 
large group of students who adapt to our mold dutifully or resign- 
edly. 

In this context of diversity, realizet' in some quarters, potential 
in others, the Colleg.=* Board has bee:i trying to make some head- 
way. A year ago you heard Richard Pearson, the President of the 
College Board make this proposition: 

Any other individual (and here he was speaking of those individuals 
who do not progress directly from high school to a two or four year 
college), of whatever age, shouU expect opportunities to demonstrate 
throu^ independent study or other off-campus learning experience 
that he is qualified fw formal study at the college level® 

Let me describe the extent of the Board's commitment to this prin- 
ciple. Seemingly this informative aspect of my comments should be 
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the easiest For here I can rest with confidence on the wisdom and 
energies of those College IBoard and ETS staff members who have 
made ihe College Level Examination Program a reality. But 
reflection about the scope and daring of the venture alone could 
consume all the time allotted for this presentation. Thus the infor- 
mation I offer is sketchy and hardly does justice to the intentions 
of the College Entrance Examination Board which sponsors the 
program. 

CLEP, the inevitable acronym for the College Level Examination 
Program is essentially a national system of placement and credit 
by examination. I stress this essentially for two reasons: 1, the ex- 
planation abbreviates considerably the diverse uses of the pro- 
gram and 2. it may generate unwarranted suspicions about its 
purposes. The major objectives of the Program attest to its real and 
potential diversity: 

1. To provida a national program of examinations that can be used 
to evaluate nontraditional college-level learnings, specifically 
including independent study and correspondence work; 

2. To stimulate colleges and imiversities to become more aware of 
the need for and the possibilities and problems of credit by 
examination; 

3. To enable colleges and universities to develop appropriate pro- 
cedures for the placement, accreditation and admission of transfer 
students; 

4. To provide colleges a'ld universities with a means by which to 
evaluate their programs and their students* achievement; 

5. To assist adults \i3io wish to continue their education in order to 
meet licensing requirements or qualify for higher positions. 

In many ways, CLEPs b'^ginning, which dates back to 1965 when 
the College Board's Council on College Level Examinations was 
appointed and began the Program's development, has been aus- 
picious. First, the Program is already operational because it rests on 
the College Comprehensive Tests, initiated and developed by ETS, 
now sponsored by the College Board as the first phase of CLEP. 
The original concern and investment by ETS in no small measure 
accounts for the positive response to the Program. Second, CLEP 
has enjoyed the generous support and financial assistance of the 
Carnegie Corporation. In addition, it has the endorsement of the 
Federation of Regional Accrediting Commissions and the Commis- 
sion on Accreditation of Service Experiences. Finally, the Program 
begins with strong leadership by some colleges and universities and 
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with at least attentiveness on the part of a great many more institu- 
tions. 

Today CLEP consists of two types of examinations. The General 
Examinations which are designed to provide a comprehensive meas- 
ure of undergraduate achievement in five basic areas of the liberal 
arts-English composition, humanities, mathematics, natural sciences 
and social sciences (history). They are intended to measure an indi- 
viduals knowledge of fundamental facts and concepts, ability to 
perceive relationships and understanding of basic principles. The 
five examinations taken as a battery include those subjects typically 
required of general education students in the first two years of 
college. The Subject Examinations, and there are currently 11 of 
them, are end-of-course tests appropriate to courses taught in the 
undergraduates years on either a semester or yearly basis. 

Faculty members from colleges and universities throughout the 
country serve tlie College Board well as the examining committees. 
With the assistance of test specialists from ETS they develop the 
examinations and perform the consistent review which ensures the 
appropriateness of the program to the collegiate enterprise. 

The future of the College Level Examination Program rests 
heavily on three things: One, the vision and capacity of the College 
Board to expand the program when and where appropriate, while 
maintaining the quality of its initial offerings. And that expansion 
has already begun. By 1968 five new subject examinations will be 
available-marketing, money and banking, statistics, history of 
American education and educational psychology. Further, exami- 
nations will soon be available not only at institutions with which 
an individual contracts, but also at national testing centers, as is 
the case with the admissions testing program. Two, the future of 
CLEP depends upon the evaluation of the present use of the Pro- 
gram. Let me cite just four examples of current usage to indicate 
the diverse purposes which CLEP may serve and to illustrate the 
point that assessing intellectual achievement on a national basis 
does not necessarily stifle, threaten or usurp the prerogatives of 
individual institutions. 

The University of Iowa, one of the country's major state universi- 
ties, has recently authorized the use of the College Level Exami- 
nation Program's General Examinations as an alternative way for 
students to meet graduation requirements. That is, students may 
meet requirements either by taking approved courses or by exami- 
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nation. Montgomery Junior College, a public institution in Mary- 
land, uses the General Examinations to provide an indication of 
how well its graduating sophomores perform in comparison with 
the national sophomore normative data and to provide a counseling 
tool for working with their graduates. In the School of General 
Studies at Queens College of the University of New York, the 
General Examinations are being used to assist with the placement, 
screening and admission of adult students. San Diego State College 
participates in CLEP through its use of the examinations to insure 
basic levels of achievement in the liberal arts area of secondary 
education candidates who seek to enroll in student teaching. In 
these examples, and others, the institutions decide not only how the 
exams are to be used, with or without credit provisions, but what 
standards of performance are acceptable. 

Stfll, the future of CLEP depends to a great extent on the will- 
ingness of colleges and universities to react to the Program and to 
shape its development. The Program at times seems embarrassingly 
simple. It presupposes no particular pattern of preparation nor 
expects any common background. It provides an alternative for 
recognizing and rewarding achievement and for encouraging stu- 
dents to continue an education which at times doesn't appear to 
conform to traditional patterns. 

I began with Spring training; let me close with three quick pitches: 

1. If we continue to operate in a vast ignorance of the true di- 
versity of students and potential students in this country we engage 
in shallow investments and superficial concerns which degrade the 
entire educational process* 

2. The alternatives available to the student, typical or atypical, 
reflect little comprehension of his individuality and precious little 
regard for what learning is all about. 

3. Programs like the College Level Examination Program are 
designed to serve human beings and diversity. If they solidify into 
one more obstacle to an individual's self-realization, we aren't 
doing our job and neither are you. 
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Citizenship Assessment 



VINCENT N. CAMPBELL 

I think I win the prize for the least inspiring title on the program. 
But let me try to compensate slightly for that by saying that I want 
to discuss^ not just citizenship assessment, but the national assess- 
ment effort as a whole. I think more of you are familiar with the 
whole enterprise than you are widi any particular subject matter or 
particular part of the assessment effort. 

About $40 billion are invested annually in American education, 
and a good question is: What kind of evidence do we have that it 
is achieving what we really want it to achieve? This is an especially 
appropriate question in view o.^ the fact that we are now in the 
midst of a flurry of educational innovation, and there seems to be a 
general feeling that things need changing. Teachers, school admin- 
istrators and other people concerned may have pretty good intuitive 
feelings about what needs improvement, but in view of the size of 
the enterprise and the importance of the outcome, it seems risky to 
rely on these intuitive feelings as a way of deciding whether edu- 
cation is achieving what it is supposed to be or not. 

As I see it, the purpose of the national assessment is to measure 
specifically what important achievements are actually being reached 
and which ones we think that are important are not being reached. 

The national assessment is an idea whicft has not yet gained full 
acceptance, as many of you are aware. The whole program is 
divided into two phases. In the first or exjiloratory phase, now nearly 
completed, the objectives of assessment were developed and the 
first draft instruments to measure these objectives were prepared. 
After these are revised and reviewed thoroughly, if it appears that 
the assessment is ready to proceed, students from all over the 
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country wil! be sampled and measures of dieir achievement in eight 
major areas will be obtained. These areas are: citizenship, voca- 
tional education, social studies, the sciences, mathematics, reading, 
music and art. 

NIany educators have asked why such an elaborate enterprise is 
needed to measure the achievements of education. Why not look at 
ordinary classroom testing results or use existing standardized tests. 
The answers are straightforward. Ordinary teacher made tests vary 
a great deal from school to school, both in content and in quality 
and they are seldom based upon a careful specification of the ob- 
jectives which students are meant to achieve. Another objection is 
that many tests are predominantly tests of memory. When we ask 
teachers what student achievements are most worthwhile they claim 
that memorization is not that important, yet memory is mainly 
what they are testing for. Ri^t there is a mismatcii that needs some 
attention, and maybe the national assessment will help point this out 

As for standardized tests, the main failing is that these are not 
designed to assess the achievement of a group on specific important 
objectives. They are designed rather to discriminate among indi- 
viduals. Important achievements which nearly everyone masters, or 
which hardly anyone does, are purposely left out. But one purpose 
of the national assessment is to decide what objectives are being 
achieved by nearly everyone and what objectives are being achieved 
by very few people, as well as those which are being achieved by 
about half of the students in the country. 

Existing tests and data, then, were considered inadequate for a 
national assessment. The first step in the plan therefore was to design 
objectives and develop new measuring instruments to measure 
achievement of these objectives. The contractors which have been 
developing these instruments are the Educational Testing Service, 
the American Institutes for Research, the Pyschological Corporation 
and Science Research Associates. 

Now let me describe briefly what I see to be the main features 
of the national assessment as a whole. The main purpose is to obtain 
accurate information on the achievements of the nation as a whole 
and major subgroups widiin the nation as a basis for better planning 
of public policy and research. 

Second, only those achievements agreed to be important by 
scholars, schools and laymen are being assessed. There is no impli- 
cation here that achievements not being assessed are not important 
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It is solely that in order to make the best use of the money available 
the idea is to assess objectives which all three groups; schools, 
scholars and laymen agree are important. 

Third, each student sampled will take only a small fraction of the 
measures. No score for an individual is wanted, and the total num- 
ber of measures required to thoroughly assess all tliese specific 
objectives is very great. Perhaps 50 or 100 hours of assessment in 
all, or more. It would be unfeasible to expect one student to take 
all these measures. Therefore each student's participation will be 
limited to one hour or less. 

The target age levels for the assessment are ages 9, 13, 17 and 
adult. A new assessment is planned to take place every few years 
and the objectives and measures will be updated and revised before 
each new assessment. The purpose of repeated assessments, of course, 
is to give an idea of the progress that is being made over the long 
course. 

And last, the results will be reported in a form that is understand- 
able to laymen rather than as abstract test scores. For example, we 
are not trying to get a citizenship score of 40.6 with a standard devia- 
tion of something or other for a person or even for the nation as a 
whole. Rather, for each measure of an objective we want to know 
what percent of the population is achieving it. It might be reported, 
for example, that 10 percent of the 17-year-old urban boys in the 
Midwest have written a letter to a public figure about some specific 
issue. In such terms we feel we can communicate to all the people 
who are concerned about education* They don't have to guess at 
what went into a test score. They can see the measures, or at least 
samples of the measures on which results are reported. 

Several people have asked why citizenship is included in the 
assessment program since it is not a traditional discipline in the 
educational enterprise. The reason is that many important goals that 
education could achieve are not included in the traditional subject 
matter disciplines. People agree that citizenship is a worthwhile 
thing, but it tends to get relegated to limbo, the only vestige being 
a global rating by teachers at the end of six weeks. It really receives 
very Uttle attention in ihe educational enterprise. With the excep- 
tion of a few schools, there is not a strong educational effort focused 
on citizenship. 

Also there has been very little specific assessment on a broad scale 
of civic achievement either in education or outside. There has been 
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even less in the basic disciplines of mathematics, science and social 
studies. The few careful efforts at citizenship measurement that 
have been made have occurred in experimental civic education pro- 
grams such as those in Detroit and Kansas. 

To many people evaluating citizenship probably calls to mind 
giving a student some sort of over-all rating. I want to make it clear 
from the start that that is not the approach here at all. We are not 
saying how good a citizen Johnny is on the whole. We are instead 
constructing measures of how well specific behavioral objectives in 
the citizenship area are being achieved by large population groups. 

The first stage of our procedure was to decide what the most 
important citizenship objectives are-what do schools, scholars and 
laymen want students of various ages to accomplish? Next, we 
developed specific exercises to measure the extent to which each ob- 
jective is being achieved. For each objective a rationale was written 
describing why certain kinds of measures seemed appropriate and 
why other kinds seemed inadequate. The purpose of the rationale 
was to set forth our reasoning for inspection, not only by ourselves, 
but by other people who were concerned with the quality of the 
measures. Also if a measure wasn't working out another could be 
substituted for it on the basis of the rationale. 

Our standard for choosing citizenship objectives was their im- 
portance to consensual civic goals in our society. Whether the objec- 
tive is being achieved well or poorly, now, and how much schools, 
home, church or other institutions have contributed to the present 
level of achievement, were not considered at all. We did not examine 
the current curricula of schools in the area of citizenship. Nor did 
we choose the objectives which would be easiest to measure, though 
it is a very tempting thing to "search where the light is best" when 
you are working in a difficult area. Consequently, importance to 
society, as agreed upon by teachers, scholars and laymen, was the 
sole basis for selecting objectives. 

In order to keep to the heart of the matter, some areas related to 
citizenship were not included in our objectives. One class of objec- 
tives excluded was those which are clearly more central to some 
other subject matter area being assessed. For example, "knowledge 
of ways of life in other cultures,** we assumed, would be covered in 
social studies. "Religion,** living a rich, full life** and "vocational 
competence'* which are important in a well-rounded life were 
judged to be only tangentially related to citizenship. So by logical 
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sequence, only those aspects which were judged to be of particular 
civic relevance were included. For example, we measure freedom 
of religion as a constitutional right because it was of particular civic 
relevance, but religion as a whole was not covered. 

We started the process of choosing objectives by reviewing pre- 
vious lists of citizenship objectives and boiling these down to one 
parsimonious but comprehensive list of about twenty objectives. 
The failing of many writers of educational objectives in the past, 
we thought, had been that the objectives were confined to glowing 
generalizations which did not have much effect on anything. In 
order to get at the crucial specifics, we took two steps: 

First, outstanding local teachers familiar with each target age 
group worked for weeks with our staff to break down each general 
objective into the most germane behaviors appropriate as goals for 
a given age group. Tliey drew on examples from their own ex- 
perience and compiled a list of very specific behaviors beneath each 
general objective. 

And second, students and adults in each age group were asked 
to recall and describe outstanding citizens of their acquaintance and 
to recollect specific incidents and descriptions. These incidents and 
descriptions, about a thousand in all, were used to check the com- 
pleteness of our initial list of objectives. 

The revised list of objectives, each broken down into important 
behaviors, was then worked over for three days by a panel of na- 
tional leaders in citizenship, education and related social sciences. 
Persons in varying community roles also reviewed our objectives 
and made comments, and the objectives were further revised. Some 
of these community consultants were public and private school 
administrators, counselors and teachers, a judge, a county planner, 
social scientists, labor and business leaders and others. The final list 
was submitted to the exploratory committee directing the whole 
assessment program, which is chaired by Ralph Tyler and has Jack 
Merwin as its current staff director. The committee held meetings 
with panels that were made up of concerned laymen from all regions 
of the country for the purpose of critically reviewing the objectives 
submitted by the contractors in all subject matter areas. A few 
wording changes were recommended by the laymen, but they did 
not recommend elimination of any of the citizenship objectives 
which are listed in abbreviated form below for seventeen year olds. 
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CITIZENSHIP OBJECTIVES 

Acts of good citizenship by 17-year-olds: 

I. Show concern for the welfare and dignity of others. 

They respect the worth and individuality of all persons regardless 
of religion, beliefs, race, or other social or personal characteristtcs. 
They weigh the effects of their own actions on the health and welfare 
of others. They seek to better the living conditions of less fortunate 
people, including people who are denied equal socio-economic op- 
portunity. They help other individuals in need. They are loyal to 
country, to mankind, and to other groups whose values they share. 

II. Support constitutional rights and liberties of aU individuals. 
They recognize when consHtuHonal rights are being properly ex- 
ercised or dera>d and defend those rights, regardless of the victim s 
unpi>pularity. They understand the values of our traditional freedoms 
and due process of law. 

III. Help maintain law and order. 

They understand the need for law and try to keep informed of the 
law. They comply with the law and school regulattons, or when th^ 
think a rule unjust, they oppose it by lawful democratic means. Their 
own behavior is ethical, and they encourage ethical behavior in 
others. 

IV. Know the main structure and functions of our governments. 
They understand the reasons for having government and the con- 
stitutional source and limitation of our governments. They know the 
main powers of each branch of government, and the relations among 
levels in om federal system. They value poliHcal ojpposition and see 
the ne-d for having a vanety of interest groups. They recognize the 
impor.ance of citizen participation in government and the many ways 
in which a citizen can influence public policy. 

V. Seek school and community improvement through active 
democratic participation. 

They believe that each person s civic behavior is important and 
each should do his part to help solve society's problems. They take 
an active part in groups which seek to better the school or communi- 
ty. They help achieve group goals by cooperating with other group 
members, and they support democratic procedures in group meetings. 

VI. Understand problems of international relations. 

They seek world peace but are aware of the many causes of inter- 
national conflict and of dangers to national security. They favor con- 
structive efforts to increase understanding and accommodation among 
conflicting nations. 

VII. Support rationality in communication, thought, and action on 
scnool and community probk is. 

They seek out and critically in erpret irformation of civic impor- 
tance. They try to understand and wei^ the ments of alternative 
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viewnoints and actions. They are effective problem-savers and plan- 
nS^ht ^pport education and free comnunication, and commu- 
nicate their own views clearly and honestly. 

VIII. Take responsMity for their own personal development and 

Theri" maximum use of opportunities to furtfier their mvn 
edSaXon S prepare for adult rol^. They show init,at,ve and ddi- 
gence, and fulfill voluntary commitments. 

TX Helo and respect their own families. 

T?ey ,4pectSeir parents' wishes and the views of aU family mem- 
ber Th^aTd the civic development of younger sibbngs and he^ 
S W duties. They discuss and relate peer group influences to 
family values. 

The nu.e main objectives represent "clusters" of related unnortem 
civic behaviors. The order of listing the nine is not hieraichical and 
tes no special meaning. The list is quite similar for the other age 
^els However, as each objective is broken down into more spe- 
cTc behaviors Aey are different a^ the different age levek It is 
only Tthe general level that there is considerable similarity We 
e«Lft a more complete description of the objectives to be available 
tSushed form in the not-tJo-distant future after the exploratory 
committee has had a chance to pass on them. 

Please keep in mind that these objectives are intended solely as 
a working guide for the difficult task of assessment. We are not pre- 
Sng how an individual citizen should behave or how he should 
7m day. In fact, no one person could be expected to perfoim 
aKe speic behaviors we have included. Some ach.evemen^ 
suci as sLking public office or raising children well will naturally 
o^y I attempt^ by certain citizens. But since Ae assessment i 
intended to describe the achievements of a wide population of 

citizens, not individual r "° P^w"!' started as 
Let us now turn to the achievement measurement. We started as 

I explained, by writing rationales in order --^^^.^^^f.^^^^^^^^^^ 
developing these measures explicit and reviewable. After we had 
p ept^ L draft measures they were submitted to two kinds o 
fests. First they were tried out with a few^dents, tiie nun^ber of 
tudents depending on the nature of the me^ure. And second they 
were revie^Wl aglin by committees of laymen from around the 
country to see to what extent they were objectionable. 

S of the measures which we have developed m the area of 
citizenship are intended to determine what citizens do or have done 



20 VINCENT N. CAMPBELL 



rather than whether they know what civic behavior is proper or not 
For some objectives, we do seek knowledge as the achievement 
sought, but often where knowledge is assessed it is taken as an indi- 
cator of past behavior rather than as knowledge ^or its own sake. 
For example, if we ask st\idents to name a favorite book or game of 
their younger brothers or sisters, we are not valuing this knowledge 
in '*^sclf, but iisins it solely as an indication of how much attention 
they give to young jr family members. 

'^"^e think that nost important citizenship objectives cannot be 
assessed adequately by paper-and-pencil techniques alone. Inter- 
views will be used ex -nsively, especially with nine-year-olds< In fact, 
in the citizenship area we have not proposed any paper-and-pencil 
measures for nine-year-olds. In order to assess reactions to spoken 
messages, we are having some of the older students listen to tape 
recordings and then respond to these communications in certain 
ways. Another type of measure we are using is recall by teachers 
and students of specific incidents of certain kinds such as helping 
other students and school assignments completed or not completed. 

Direct observation of behavior in groups is an expensive measure- 
ment technique, but it was our conclui ion that it is the only valid 
way to assess certain objectives in the area of citizenship. For ex- 
ample, we could think of no less expensive way of measuring "effec- 
tive cooperation with others to achieve a group goal" than by ob- 
serNnng people in their interaction and recording certain specific 
behaviors observed. Another objective of this sort was the willing- 
ness to express one's views in public. A third objective was the 
respectful treatment of other individuals. For each school age level 
we therefore contrived one or two standardized group situations in 
which these and other behaviors could be observed. Any measures 
which we have developed involving direct observation of behavior, 
judging by our tryouts to date, will require the use of a checklist 
to simplify the coding of the observations and at least an hour or 
two of training for each observer. 

The tryouts are now underway for all subject matter areas in the 
national assessment, and the purpose of these tryouts is mainly to 
establish administrative feasibility. To some extent, the tryouts also 
help us in detecting our mistaken premises and other weaknesses in 
the measures. But mainly we want to know how the measures should 
be administered and who should do it. That is, what things can 
teachers in the schools themselves handle adequately and which 
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tasks require more expert administration. Also, how should the 
measures be packaged, that is, how many different subject matter 
areas or formats can a student respond to adequately in a half an 
hour. 

Late in 1967 more extensive tryouts will be conducted to deter- 
mine whether we are truly sampling all levels of difficulty. That is, 
are we getting measures of what 90 percent of the students can do 
and what only 10 percent can do, as well as of those achievements 
reached by about half the age group. 

These tr)'Outs are not validity studies in the ordinary sense. I am 
not sure whether there is any group on which you could validate 
gains for measures of this sort. The main criterion for the F.cceptance 
of any measure is, whether on inspection of the measure itself and 
I the rationale for its use, one is convinced that the measure validly 
measures achievement of the objective it is intended to assess. 

Before closing I should mention something of the controversy that 
has surrounded the national assessment. "Fear of national control" 
seems to be the theme brought up most. I have tried to understand 
this somewhat, but I must confess that I don't, and this is certainly 
not because I favor national control. On the contrary I am for local 
control of education. I am not even for state-adopted textbooks, as 
we have in California. The purpose of the national assessment is to 
gather information about the achievement of objectives. How this 
information 'is used is entirely up to the person who is using it. It 
seems to me that there is less reason for teachers to prepare a student 
to do well on a national assessment, since the chances are one in a 
thousand that the student will be involved in the assessment any- 
way, than there is for a teacher to prepare students to do well on, 
say, college entrance examinations which have considerably more 
effect on the individual lives of students. It doesn't seem to me that 
educators in general are exerting great effort to adapt their curri- 
culum to the tests that now exist. So I don't understand exactly why 
it should be expected that the national assessment, which will not 
even characterize any indi"- lual student, should be feared on these 
grounds. 

Another point that I would like to make is that in future national 
assessments these objectives will be updated so one need not fear 
that the objectives are going to become archaic and that a fixed set 
of measures is somehow going to determine the future course of 
education. Obsolescence does not seem to be a valid criticism of the 
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plan. We hope that national assessment will, on the contrary, peri- 
odically provide a current, accurate appraisal of our society's aspira- 
tions and its degree of success in achieving them. 



Educational Criteria 
and the Chesire-Cat 



SCARVIA B. ANDERSON 

You will recall that the baby had just turned into a pig-rather a 
handsome pig though it would have made a dreadfully ugly child— 
and trotted off into the wood. Alice was startled to see the Cheshire- 
Cat sitting in a tree. 

After some preliminaries, she ventured to inquire, *Would you 
tell me, please, which way I ought to go from here?** 

"That depends a good deal on where you want to get to," said 
the Cat. 

"I don't much care where—" said Alice. 
"Then it doesn't matter which way you go," said the Cat. 
"-so long as I get somewhere,** Alice added as an explanation. 
"Oh, you're sure to do that," said the Cat, "if you only walk long 
enough."^ 

There are many Alices among educators and would-be evaluators 
of what goes on in education. We are seeking direction, but we are 
not sure (and sometimes say, defensively, we don't much care) 
where we want to go— and our explanations are less than satisfactory. 

But perhaps we are not entirely to blame. The objectives that we 
seek and the criteria by which we will know whether we have at- 
tained them are as elusive as the Cheshire-Cat. They keep appearing 
and vanishing so suddenly that they make us giddy, as Alice was. 
Even when the Cat obligingly disappeared quite slowly, beginning 
with the end of the tail and ending with the grin, you will rec^l that 
the grin remained some time after the rest of it had gone. And that 
was quite disconcerting to Alice, just as it is disconcerting to us to 
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realize that we have sighted some element of our criterion but the 
rest still eludes us. 

Of course, in talking about educational criteria I am also talking 
about educational objectives, because the two are inextricably inter- 
twined to the degree that an adequate explication of the criteria 
makes almost' unnecessary any separate statement of the objectives, 
and an inadequate explication of the objectives makes almost im- 
possible any definition of the criteria. 

In this mood of shifting some of the blame for the situation we 
are in from ourselves to those ornery rascals, the criteria, let us look 
at some of the reasons why the criteria make themselves so hard to 
get hold of. 

First, as Henry Dyer from the ETS branch office in Princeton says, 
"criteria rely too much on the magic of words.'*^ An objective 
pompously introduces itself as beaming how to live and work to- 
gether."^ Now by what criteria would we know whether children 
had learned to live and work together? 

Dyer goes on to mention two other reasons why educational goals 
have been nonfunctional in the past, and I take "nonfunctional" to 
mean unproductive of adequate criteria. One is that there has been 
too little public participation in formulating them; the other is that 
there has been too great a readiness to suppose that they are already 
given and require only to be achieved. 

A fourth source of problems arises out of the field with which 
many of us here are most closely associated: measurement. Some 
criteria appear momentarily and then disappear because there are 
no existing, immediately available means of measuring them. In 
some cases, this discourages us; in others, we grasp our old security 
blanket statement, "Many of the things I teach are intangible and 
cannot be evaluated."^ 

What can be worse is that, if there is an existing measurement 
instrument, we are inclined to assume that there is a worthwhile 
objective to go with it Thus, because we just happen to have a test 
which purports to measure behavior X, we suddenly decide that 
student competence in X was something we had very much desired. 

Those of us who talk of educational criteria are sometimes accused 
of having some of the characteristics of the criteria we criticize. We 
tend to speak in vague generalities. Lest I fall completely into that 
trap, let me describe some of our experiences in searching out one 
particular criterion which has proved especially difficult and elusive.^ 
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We are not sure of the proper name for it, but it has been called 
"school readiness," and it is something which most preschool pro- 
grams say they are trying to develop and most first grade teachers 
decry the lack of in some of their pupils. (We shall not concern our- 
selves right now with the facts that some psychologists question the 
viability of "increased readiness for public school" as a major goal 
of preschool stimulation programs^ and that most teachers admit 
that their job is one of coping with all children who come to them, 
regardless of whether they are "ready.") "School readiness"— at any 
rate the words are with us. Tests have the words in their titles, 
teachers write the words on their reports about children, and spon- 
sors of picschool programs list them in their proposals for funds. 

We used two kinds of detection devices as we went hunting 
"school readiness." They are both rather old-fashioned. The first is 
the "critical incident" technique; the second, the group discussion. 
The first detection device was activated last fall with the help of 
a national sample of about 300 first grade teachers. They were asked 
to do the following things: 

1. List all of the children in their classes and tell us when they were 
horn and whether they were boys or girls. 

2. Answer the following question about each child: As of this date, 
how ready do you think this child is for the first grade experience? 
—Ready in all respects? Ready in most respects? Ready in only a 
few respects? Not ready? (We had originally considered a three- 
point rather than a four-point rating: ready, not ready, and can't 
judge. But the group of teac^iers on whom we tried out the ques- 
tionnaire indicated— very intelligently— that very few children 
were wholly ready or not ready. Tney were also unwilling to 
admit that they couldn't judge any child!) 

3. For each chila rated ready in all or most respects, give a specific 
example of one thing which he did or said which suggested! that 
he w(us leady. Or, alternatively, for each child rated not ready or 
ready in only a few respects, give a specific example of one thing 
which the child did or said which suggested that he was not 
ready. 

At a later date, teachers also supplied us with information about 
children's preschool experience and home background. 

The some 7,000 incidents which teachers provided have been 
classified into ten content categories. The content categories and the 
percentages of incidents recorded under each are given in Table h 
Let's look at the first column of percentages there. 

If we assume that what teachers choose to list is somehow related 
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to what they value, we can conclude that these first grade teachers 
value the intellectual over the nonintellectual, verbal skills over 
quantitative skills, and personal emotional development over peer 
relationships. We note too that conformity to classroom procedures 
(our category closest to old-time "conduct") is still highly valued by 
teachers — or at least problems concerned with 'lining up," "not 
talking out of turn," and so on are cited more often than problems 

TABLE 1 

"Readiness" Incidents CriED by First Grade Teachers 
(No. of incidents = 6708) 

Percentage of All Percentage of Incidents 
Incidents Positive Negative 

INTELLECTUAL CRITERIA 

Verbal skills and understandings 

Quantitative skills and understands 

Graphic skills 

Performing arts skills 

General intellectual functioning 
NONINTELLECTUAL CRITERIA 

Attitudes toward school and 
school work 

Conformity to classioom procedures 

Personal emotional development 

Peer relatonships 

Motor coordination and 
physical condition 
NOT CLASSIFIED 



in motor coordination or quantitative or graphic skills. ("Graphic 
skills" by the way includes painting, drawing, making letters-the 
execution of lines or marks on a surface.) 

The last two columns of Table 1 show the percentage of positive 
and negative incidents in each category. As would be expected, 
there were more positive than negative incidents cited, simply be- 
cause the teachers marked more children ready than not ready, and 
the rules of the game specified that they were to substantiate their 
ratings by incidents. (The actual ratio was 3 to 2.) What was not 
quite expected was that teachers gave proportionately many more 
positive incidents in the intellectual than affective areas. In other 
words, good social adjustment and poor cognitive development were 
less pr<Kluctive of teacher incidents than their converses* This is a 
puzzle and much more exploration of the data is needed. (Inci- 
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dentally, these data are very hot oflF the computer and almost as 
strange for me to contemplate as for you.) 

The frequencies with which incidents were cited are being further 
analyzed in terms of characteristics of the pupils who were rated, 
and characteristics of the locations in which the ratings took place. 
In other words, we are entertaining the hypothesis that the criteria 
of school readiness may be diiSFerent for teachers with diflFerent 
amounts of training and experience, in cities of different sizes, and 
for boys versus girls, younger children vei^us older children, children 
from different socio-economic backgrounds and amounts of pre- 
school experience, and so on. 

Our second detection device, the group discussion, was one we 
are more accustomed to using, especially as we define test specifica- 
tions (which are very close to educational objectives and criteria, 
we hope). A group of child development experts^ met in Princeton 
to tackle the problem: What should be the characteristics of a com- 
prehensive set of instruments designed to assess children's readiness 
to enter usual school programs? Tliey were asked to try to specify 
the categories of behavior which should be included and to attempt 
to weight the categories in terms of importance. 

The committee, under the leadership of Dr. Miriam Goldberg of 
Teachers College, took an interesting tack in their deliberations: 
They tried to outline tiie kinds of tasks and situations with which 
children would be confronted during their first few days in the first 
grade and then reJate these to the kinds of abilities, skills, etc., 
children would have to have to cope successfully with the tasks. 

You will be interested to know that the committee was unwilling 
to ascribe anything but equal weights to characteristics in the three 
major categories: sensory/ motor, cognitive/ intellectual, and social/ 
personal. They felt that measurement of physical characteristics 
should best be left to appropriately trained medical personnel, but 
that criterion instruments could be developed which teachers could 
use to assess 9 cognitive/ intellectual areas, 10 social/personal areas 
and 4 areas wh^'ch cut across two or more of the other main areas. 
The major categories which they produced are given in Table 2. 

Of course, to qualify ss useful criteria of "school readiness,** each 
of these categories has to be spelled out in more specific terms. An 
example of a first stage in such spelling out is given in Table 2 for the 
crucial cognitive category, Translation from one code to an- i^r. 
The next stage will involve identification of what letters and S£a\ At 
sounds are to be discriminated, and so on. 
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It has been of interest to compare the incidents cited by first 
grade teachers with tiie categories developed by the assessment 
committee. The overlap is considerable, with the teachers providing 
some rather dear examples of what the committee had in mind, and 
thus direct help in spelling out the committee s recommendations 
in more specific terms* 

For example, a teacher^s ''Expresses ideas in complete sentences" 
conveys some of the sense of the committee's Language output 

Negative and positive instances of Concept development and for- 
mation are provided by these two teacher statements: 

In a series of four houses, she could not locate one that was different 
(The door and window were ia revere positions.) 

When ve cut a pumpkin for a jack-o-lantera, he told us to save the 
seeds to plant next spring so that we coidd have a pumpkin for next 
Halloween, 

Positive examples of the category Translation from or^ code to an- 
other are provided by these three teacher instances: 

Can interpret the feelings of picture story characters by observing 
their f acisu expressions. 

Counted his lunch money correctly the first day of school. 

Delighted when he was able to recognize some of his reading words 
in books at home; said he especialty luced the word took. 

Negative instances of the same category include these: 

Had di£BcuIty distinguishing one letter name from another. 

Writes her name with all capital letters, most of the time backwards. 

The child about whom the teacher wrote "Unable to recall what 
comes next in a story that has been read several times** may have 
difiBculty with Language comprehension and Ability to absorb, store, 
and retrieve information* 

On the social/personal side, we believe that the child who *Went 
home one day at recess, thought school was out** was having some 
trouble conforming to and remembering school routirws. "Waits 
for her turn** seems to say something about Fitting into the group 
and the pecking order, and the child who "Hums to himself con- 
tinually** seems to be lacking in Impulse control. 

So far we have involved several hundred people and many thou- 
sand dollars in this attempt to pin down 'school readiness,** and, as 
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you can see, we are still not done. We are coping first with the 
''magic of words" problem. What eventually comes out of this work 
at what is more or less a national level, will have to be considered 
in terms of its relevance for local school systems and specific edu- 
cational programs. And there are still the measurement problems to 
be solved. For example, we are not content to say that Tolerance of 
ambiguity is either too intangible to measure or that satisfactory 
ways of measuring it have been developed. 

The present project simply serves to Ulustrate what a tou^, time 
consuming job it is to specify the things we want to accomplish with 
our educational programs, and to define the criteria by which we 
shall know whether we've accomplished them. The alternative, of 
course, is to continue going along, hoping we will get somewhere 
if we only walk long enough. If we take the alternative, however, 
how are we going to hold up our heads and preserve any sanblance 
of professional integrity as we are bombarded with the sensible 
questions asked by a sensible public? *'Where is somewhere and 
how long is long enotigft?— with Head Start, with new elementary 
and secondary curricula, with the ungraded school, with the lan- 
guage laboratory, with computer assisted instruction, and with all 
the other educational innovations which are clamoring for enormous 
amounts of support 

In case you think it has been a little frivolous of me to lean so 
heavily on Lewis Carroll in this speech before a distinguished edu- 
cational audiafice^ let me read you five quotations: 

L We had the best of educations-in fact we went to school every 
day , , . with extras , . . French, music, and washing-extxsL 

2. . . . Excellent at slapping but deficient in resting . . . 

3. Robert needs to feel more 5ure of himself before he can setUe 
down to the more academic aspects of kindergarten . . . 

4. Reeling and Writhing, of course ... and then the different 
branches of Arithmetic— Ambition, Distraction, Uglification, and 
Derision. 

5. , . . Deficient at cutting and pasting . . 

Two of these quotations are from Alice in Wonderland, the other 
three from kindergarten teachers* reports to parents. Now I ask you 
which is from which? 



EDUCATIONAL CRTTERU AND THE CHESHIRE-CAT 31 



REFERENCES 

1. Carroll, Lewis. Alice's Adventures in Wonderland, New York: The Modem 
Library (Random House). P. 86. 

2. Dyer, Henry S. "The Discovery and Development of Educational Goals," in 
Proceedings of the 1966 Invitational Conference on Testing Problems, Prince- 
ton: Educational Testing Service, 1967. Pp. 12-24. 

3. Fuller, Elizabeth M. What Research Says to the Teacher: About the Kinder- 
garten. No. 22. Washington, D. C. National Education Association, 1961. 
Pp. 7, 16. 

4. Mager, R. F. Preparing Objectives for Programmed Instruction, San Fran- 
cisco: Fearon Publishers, 1961. P. 44. 

5. This project is being carried out at Educational Testing Service by Scarvia 
Anderson, Ann Juneeblut, George Temp, and Donald Trismen. A complete 
report will be published later this year. 

6. For example: Kohlberg, Lawrence. "Early Education: A Cognitive-Develop- 
mental View." Unpublished multigraphed manuscript, 1967. 

7. Miriam L. Goldberg, Columbia University; Allan Hartman, Eastern Regional 
Institute for Education; William 0. Jeiudns, The City University of New 
York; Gertrude Justlson, Howard University; Harry Levin, Cornell Uni- 
versity; Henry J. Mark, Johns Hopkins University; Irving Sigel, Merrill- 
Palmer Institute; Martin Spidder, U. S. Department of Health, Education 
and Welfare; Sheldon H. White, Harvard University; Ralph Witherspoon, 
Florida State University. 

8. Quotations 1 and 4: CarToU, Lewis. Op. cit. Pp. 121-122. Quotations 2, 3, 
and 5: Teachers' reports. New York Times, Section C, p. 34, January 11, 
1967. 



Are Educational Researchers 
Really Prepared to Evaluate 
Educational Programs? 

RODNEY W. SKAGER 

I am facing the problem sometimes encoimtered by a speaker after 
following a number of other speakers because there has been some 
continuity in what has been talked about today with respect to 
evaluation and innovation. I find that on at Least two points some- 
thing I am going to say has aheady been anticipated, indeed almost 
said in exactly the same way. I want to assure you that I have not 
been sitting back and taking notes so as to be able to jump on the 
bandwagon. 

This afternoon I find myself in the somewhat uncomfortable posi- 
tion of having to fulfill an obligation to speak at length on some- 
thing I do not know very much about. While this situation may not 
be entirely unprecedented in the history of educational meetings 
one does not like to contribute unduly toward turning this particu- 
lar precedent into a tradition. Perhaps the one thing that can be 
said in favor of sticking one s nedc out in this manner is provided 
by the experience of those of us who have tried to live up to the 
role of the teacher. By being continually forced to make noises with 
our mouths before rooms full of people, we usually work harder to 
make those noises meaningful than we mi^t have done left to our 
own devices. I only hope that this general principle will be operating 
this afternoon. 

During the past year I have found that many of the ideas and 
habits associated with my work have had to be re-examined in the 
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light of a new situation. While I suppose this happens to most of us 
from time to time, the particular reorientations I am being forced 
to make may turn out to be a relatively common experience in the 
future. Indeed, if this is not the case, I expect that the educational 
research community will fafl to meet a real public responsibility 
and will also miss a significant opportunity to expand basic knowl- 
edge. 

As everyone knows, massive infusions of funds earmarked for 
innovation and development are being administered to American 
education. The biggest single factor in all of this— though by no 
means the only one— has, of course, been the Elementary and Sec- 
ondary Education Act of 1965. It is probably not entirely a coinci- 
dence that national support for the try-out of new-fangled ideas and 
gadgets comes at a time when American industry has begun to get 
into the business of educational technology in a very big way. So 
there are plenty of things for the educator to buy and since many 
of the items are expensive, (he matter of evaluating their eflFective- 
ness becomes more important tiian ever. But even if educators them- 
selves were not concerned with evaluation, flie Federal Government 
is oncemed. Note, for example, tiie very significant clause under 
Title I of the 1965 Bill which requires: 

That effective procedures, including provision for appropriate objec- 
tive measurements of educational acnievement, will oe adopted for 
evaluating at least annually tiie effectiveness of the programs in 
meeting the special educational needs of educationally deprived 
children. 

I am sorry about that prose. Next time I will try Lewis Carroll. 

This clause, and others like it, have already spawned hundreds 
and wiU eventually spawn thousands of evaluation projects. At the 
Center for the Study of Evaluation of Instructional Programs we 
have already received many inquiries as to whether we would be 
interested in conducting the evaluation phase of a local project. 
Although evaluation needs have been with American education for 
some time, I diink it is safe to say diat» within the past five years, 
they have developed the potential for requiring far more in the way 
of professional resources dian tiKe research community has available 
now or in the f orseeable future. 

What I am going to discuss in die next few minutes amounts to 
an elaboration of the assertion that educational researchers as a 
group are not necessarily well equipped for the evaluation of in- 
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structional methods and practices. In case you are anticipating 
certain remarks, let me hasten to add that I am not going to bore 
you with another public lamentation about how our graduate 
schools do not offer or require enough background in research 
methodology and philosophy of science. This is most certainly true. 
However, excuses to the effect that students in curriculum adminis- 
tration or media do not need much in the way of research training 
are sheer nonsense. Everybody is in the research act nowadays. 
Once a student gets a degree and a professional title he is not likely 
to be challenged in the labors ^y doorway by a guard whose assign- 
ment is to ascertain that everyone seeking admission had Statistics 
210C. So, even though I could make vigorous noises on this particu- 
lar topic-and I have almost gotten started-it is really another issue. 
Statistics 210C may be a necessary prerequisite for research in 
evaluation but it is not a sufficient one. In fact, it is precisely some 
of the habits of thinking developed in Statistics 210C and Measure- 
ment 211C that are part of the problem. The thesis of this presen- 
tation is not that there are too few adequately trained researchers 
but rather that the well-trained, technically competent researcher 
may bring habits and values to evaluation research that are out of 
mesh with the problem at hand. 

But before I begm tdWng about the habits, both good and bad, of 
educational researchers I had better sketch in what I think evalua- 
tion is all about I have been more careful to do this sort of thing 
since the day I sat down in a certain barbershop in Princeton, New 
Jersey. When I asked for the usual crewcut tfie barber replied, 
"Well, now, let s define our terms." I knew I was in a new kind of 
town. 

The type of evaluation under discussion here refers in the most 
general sense, as a very broad definition, to the comparison of edu- 
cational practices and their outcomes against standards and/or 
against other educational practices. Embedded in this very loose 
definition are at least two of the factors making evaluation research 
a different and rather obstreperous sort of ^'animal." First, there is 
the use of the term ''standards,'* implying that values have a very 
significant role in what is studied. Now, I do not maintain that the 
selection of a problem in so-called basic research does not reflect the 
experimenter's beliefs about what is worthy of study, but evalua- 
tion research obviously implies a concern with values that is raised 
to the lOfli power. It is quiddy apparent, without elaborating on 
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the issue, that the ..latter of what such a complex phenomenon as 
instuction in the schools is supposed to accomplish poses the evalu- 
ation researcher with imposing prcMems of selection and emphasis. 
In contrast to his colleague in tae laboratory, the evaluation re- 
searcher will find it ahnost normal for people to have violent feelings 
about what he did or did not include as a criterion variable. 

Secondly, the use of th^ terms "educational practices and their 
outcomes" indicates a most significant fact. What is done is as 
much the subject of evaluation research as what happened when it 
was done. In reports on laboratory research, the c'escription of the 
experimental treatment is never the subject of the study; it is merely 
part of the introduction^Q the report in the journal. But in evalu- 
ation research it is at least half of the battle. It does no good, you 
see, to say that school A produced ^ore knowledgeable students 
:han school B if you cannot tell people what it was that the two 
schools did that was different. It is pointless to report that curricu- 
lum C produced better results than currLalum D if you cannot 
establish that both curricula were actually presented in the way you 
think they were presented. We all know that human beings can 
very easily give lip service to new procedures yet really do things 
in tfie same old way. 

I think one more thing needs to be said about the definition. The 
terms are deliberately as broad as possible, reflecting the nature of 
the potential subject matter. I have, for example, had to use the 
term "outcome" instead of the term "learning," not only because our 
criteria about what happens to students may encompass traits other 
than Mi usually characterized as '^knowledge" or "skill," but be- 
cause much of our data on outcome may not involve the student at 
al', either as individuals or in the aggregate. We have to be con- 
cerned with such diverse outcomes as change:, 'n the structure of the 
school as an institution, the behavior of teachers, or even-perhaps 
I should say especially— whether a new program costs more than it 
is worth. 

In recent years, some general notions about the particular charac- 
teristics of evaluation research have arisen. As far as I c?n tell-and 
I am a neophyte in this field-Cronbach's (1963) paper on curriculum 
evaluation^ has been the fulcrum around which these ideas have 
developed. Cronbach advanced, among other things, the radical 
notion that the traditional emphasis on evaluating final outcomes has 
been aU wrong-that is, Cronbach rejected the image of the evalu- 
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ation researcher as one who sits back until the developer turns out a 
finished product and then, entering the picture at its most dramatic 
moment, designs a critical experimental test, involving, to be sure, 
the random assignment of subject, and from his Olympian heights, 
eventually dispenses the masterful conclusion that group E per- 
formed better than group C at the .05 level of confid* -ce. Cronbach 
suggested that image was very nice, but is seriously hawed in prac- 
tice by the fact that the usual result is the retention of the null 
hypothesis rather than its rejection. On the rare occasions when the 
hypothesis of no difference is rejected, one has the feeling that some 
minor tinkering around with the curriculum or other method which 
came out most poorly might have changed the result radically. 

Cronbach's answer to what the evaluator ought to be doing is well 
known and shared to varying degrees by most people who are pres- 
ently writing about the matter. The evaluator should be concerned 
primarily with using evaluative information in a formative sense, 
that is, he should enter the picture early in the developmental phase, 
not at the end. He should observe how things are taking shape, pro- 
vide information that will illuminate decisions about adopting par- 
ticular strategies or methods, and, in general, sharpen up the 
developmental process. The idea is that the evaluator s primary 
role is to help see to it that the product eventually turned out is 
maximally effective, not just to aid in making a decision about 
adopting that product. This is a dynamic and constructive role, as 
opposed to the, at best, disinterested involvement of one who judges 
the final product against other alternatives. Perhaps I might antici- 
pate myself at this point by suggesting that the hypothesis testing 
psyche of the typical researcher-which implies an orientation to 
issues and questions rather than materials and services-is hardly 
attuned to the type oa role Cronbach has envisioned. Althou^ 
others, particularly Scriven,^ (1965) have been unwilling to throw 
out evaluation of final outcomes-and I suppose I would count my- 
self on that side of the fence-everyone these days sees this con- 
structive or formative role as at least a major function of the 
evaluator. 

There is another emphasis in contemporary theory about evalua- 
tion that will help us understand an aspect of the beast that makes 
many experimenters uncomfortable. And this point of view was well 
articulated by Guba^ (1965) in his paper on strategies for producing 
educational change* In this paper, Cuba strongly emphasized the 
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idea that evaluation research is properly done outside the laboratory 
and in the natural setting in which the program was designed to 
operate. Rather than the precise control over extraneous factors 
available in the experimental situation, evaluation research should, 
in effect, toss down tho gauntlet and dare the context to confound 
the results. After all, the context will always be there when a pro- 
gram is adopted in the schools. You might as well study it. Re- 
ferring to Barker s* interesting distinction between the intervention- 
ist role of the experimenter and the noninterventionist role of the 
observer-transducer in Barkers terms-Guba rejected the experi- 
ment as the primary vehicle for evaluation research and thus, in 
effect, said that the researcher in evaluation should be especially 
careful to avoid altering the phenomena that he is observing by 
setting up the artificial constrictions that an experiment implies. 

In reply to this view, I think it should be pointed out that even if 
we are entirely convinced that the effectiveness of educational prac- 
tices can be determined only in the natural setting, we have by no 
means eliminated the experiment, only the laboratory. It is cer- 
tainly possible to perform experiments without creating artificiality. 
Control is, of course, a greater problem in the so-called real world, 
but control approaching that of the true experiment, and certainly 
the quasi-experiment, does not inevitably require the researcher to 
change the conditions under which the instruction takes place. 
Further, if we allow the simulation study as an evaluation tool, I am 
not even certain that we have eliminated the laboratory. After all, 
if the evaluation researcher is to aid in the formative aspects of the 
development of educational programs, it is hard to see how we can 
eliminate techniques that are essentially simulation as opposed to 
''real" instruction. If we vary conditions and compare results in simu- 
lation research, we are back to the experiment in a situation that is 
to some extent artificial. 

While I am by no means willing to agree that the experiment has 
no role in evaluation, I do believe that the otherwise competent re- 
searcher will put far too much faith in its value in his initial ap- 
proach to an evaluation problem, and, even where experimentation 
is appropriate, go about things in a way that does not necessarily 
meet the requirements of a new situation. The first diflSculty in 
utilizing the experimental method in evaluation has already been 
mentioned. How valuable, really, are experiments in which you are 
uncertain as to the nature of the independent variables? Comparing 
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the learning of students under two different experimental curricula 
may, even in the face of Cronbach's doubts, show that one program 
is superior. But each program is a complex entity, subsuming many 
subsidiary variables. For the purposes of the experiment, we can 
call the entire program a variable. However, when we do this, we 
are in danger of committing the error made by the man who wanted 
to relegate the automobOe to some sort of primeval junk heap the 
first time one lost a race to the horse. All that was needed was a 
better spark plug, Instead, the whole vehicle was condemned. The 
results of an experiment pack tremendous power for establishing 
conviction, but we had better be certain that we know what it is 
we have been experimenting with. 

In the typical laboratory experiment, the researcher can deter- 
mine in advance everything that is going to happen, except, pre- 
sumably, his results. (And there are some who might say that he 
often does that as well.) In short, conditions can be standardized in 
the laboratory, contingencies are taken into ac<«unt, and the whole 
procedure can be so automatic that the experimenter attends a con- 
vention while his graduate assistant runs the experiment. Perhaps 
I am being facetious, but it is actually true that the well-run labora- 
tory experiment ought to go off like clockwork. Both independent 
and dependent variables are determined in advance and are usually 
relatively few in number. The subjects may be doing many things 
during tfie experiment, but only a limited set of variables are re- 
corded and studied. 

Now, I submit to you that the experimenter who generalizes these 
habits to the nonlaboratory evaluation situation is almost certain to 
produce a very uninformative, perhaps completely misleading, eval- 
uation study. The non-involvement of the experimenter with the 
subject matter of his study— as might be typified by the anonymous 
rat in the programmed Skinner Box— is simply an inappropriate 
model for most evaluation research. Let me hasten to say again that 
I am not saying that the experiment, even in the purest sense of the 
word, is always inappropriate for the evaluation of instructional pro- 
grams. But I am saying that some of the habits and expectations of 
people who do experiments (and train others to do experiments) 
may not be appropriate in evaluation research. 

The researcher must, therefore, resist the tendency to treat com- 
plex instructional totalities such as a curriculum or a school as if 
they were the simple, unidimensional independent variables usually 
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dealt with in the experiment Furthermore, he should realize that the 
description of just what the major instructional variables are may 
well be the most important single outcome of his research. And 
finally, the researcher should avoid the premature crystallization of 
decisions about what is and what is not to be observed and substi- 
tute, at least in part, an active and flexible involvement with the 
educational situation under study so that measures of variables, 
both dependent and independent, can be collected, even though 
their importance was not anticipated at the beginning of the re- 
search. 

Needless to say, this tradition of the hi^ly standardized data 
collection phase is not limited to the experimentalist An even 
greater offender is the survey researcher who designs and mails out 
a questionnaire from the splendid isolation of his office. His closest 
involvement with the subject matter may be a casual flip throu^ 
several pounds of computer print-out. Let me make it clear that 
there is nothing wrong with this as long as we are all-wise in ad- 
vance. But I would maintain, however, that the complexities of edu- 
cational treatments and their many-faceted effects make it very 
difficult to take everything into account in advance. 

But habits associated with the laboratory experiment are not the 
only problem. Everyone in educational research, even including the 
lofty psychometrician, is likely to have some habits or beliefs that 
do not integrate well with the requirements of evaluation. 

As an opener, I would like to suggest that there is just a faint 
possibility that we have oversold what are usually referred to as 
**behavioral goals," and here I must add the prayer that heaven 
may not strike me dead on the spot By "we" I refer to the research 
community, and by "behavioral goals" I mean something like the 
idea I heard expressed recently by a man representing an important 
governmental agency. "If an instructional program doesn't make 
any difference in the learning of students," he said, "we should 
throw it out and spend our time looking for something better." 
While a year ago I would have agreed with this assertion without 
giving the matter a second thou^t, I now feel the uncritical accept- 
ance of this statement by evaluation researchers is erroneous. For 
one thing, the focus on the word "learning" needs to be elaborated. 
I think the word "behavior" in "behavioral goals" should have a 
broader referent 

Take the ungraded school, for example. One of the things that 
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this sort of organization is supposed to generate is a greater moti- 
vation toward independent study on the part of the student. To be 
sure, learning is a goal, to some extent an ultimate goal, but would 
we not be interested in getting some measures of the manner and 
frequency of independent study activities~and I do not refer to 
anything elaborate here-as a perfectly appropriate criterion meas- 
ure? Moreover, I would be willing to let the measure stand by itself 
as being at least equal in importance to a measure of what has been 
learned. After all, independent study may tend to personalize learn- 
ing to some extent. Give a typical broad gauge achievement test to 
a classroom full of independent learners and you may miss the 
mark in many cases. Unless you are insightful in designing the 
learning measure (for example, establish the dangerous precedent 
of asking the student what he studied before making up the test), 
you may be unable to confirm the result you had expected. 

But there is another aspect of this overly restrictive definition of 
"behavioral goals" about which it may be harder to generate con- 
viction among researchers. I have anticipated, this point already by 
emphasizing that evaluation criteria need not be restricted entirely 
to measures taken on the student. This proposition is advanced 
with some misgivings, because I certainly do not want to give the 
impression that the traditional approach to evaluation based on 
long check lists about the facilities and procedures, without any 
reference as to how people are behaving, has much validity. But 
there are many things going on in the school that are highly sig- 
nificant, that can be assigned values, but which are only tenuously 
reflected, if at all, in the learning of students. For example, we 
would not have to give achievement tests to students to decide 
whether it is a good thing that a 20 percent increase in annual rate 
of teacher resignations accompanies the ungraded school. To be 
sure, we will want to check on whether we have instituted a whole- 
some process of natural selection accompanied by "the survival of 
the fittest" of our teachers. But measuring our students' learning 
seems to me to be a very indirect way of determining this. There 
are many goals that ought to be studied primarily at the uistitutional 
level. For example, determining the effectiveness of the League of 
Cooperating Schools, set up by the Institute for the Development 
of Educational Activities, involves legitimate criteria that are rather 
far removed from the behavior of the student* As an organization 
set up to stimulate innovation and change in the schools, the 
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League's efforts must be evaluated, at least in part, simply on 
whether any changes have actually occurred. For example, if it can 
be shown that teachers are meeting regularly to discuss curriculum 
innovations and if this seldom or never happened in the past, then I 
think most of us would feel that at least one positive outcome had 
been demonstrated, even in the face of the fact that students in 
the school still have the same mean score on the STEP test. Being 
broadminded about our criteria in evaluation does not imply a lack 
of intellectual or moral rigor. It merely indicates that we have re- 
covered from an inhibiting case of tunnel vision. 

With respect to criterion measures, narrow-mindedness about the 
range of variables relevant to evaluation is only one of the potential 
bad habits of the typical researcher. Considering now, the matter 
of student learning, I would suggest that there is a tendency to rely 
too heavily on standardized psychometric instruments designed for 
purposes other than the evaluation of instructional programs. I am, 
of course, referring to the achievement test. To be sure, we rely on 
the achievement test partly because measures specifically appro- 
priate to many criteria do not exist In the effort to get something 
measiured, the evaluation researcher may select his criterion instru- 
ments largely on the basis of what is available. In effect, the state 
of the measurement technology establishes the criteria rather than 
the investigator." 

But the state of the art iv not the problem referred to here. Rather, 
the difficulty is that most of us have certain fixed, though largely 
inaccurate, notions about the properties of the achievement test. In 
particular, we have the idea that such instruments are designed to 
be sensitive to what goes on in the classroom. This is untrue. The 
point has already been made by many individuals that the manner 
in which achievement tests are built actually works against this 
goal, not intentionally, you understand, but as a natural result. 
The achievement test is useful for the purpose for which it was 
designed: ordering or discriminating among individuals with re* 
spect to knowledge and skill associated with rather broadly defined 
areas of educational content. Oddly enough, evaluating people, 
which is what the achievement test was designed to do, does not 
turn out to be the same thing as evaluating instructional programs. 
We would heap scorn on an achievement test if it turned out that 
most of our students got almost all of the items right. Our legitimate 
complaint would be that the test had failed to discriminate among 
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individuals. For this reason, as well as to promote test reliability as 
traditionally defined, test makers tend to select items likely to be 
passed by about half of the subjects to whom the test will eventually 
be administered. This is fine for the discrimination application of 
achievement tests. But it is a perfectly silly way to build an instru- 
ment for determining whether we have accomplished certain in- 
structional goals. In the latter type of evaluation, we would, pre- 
sumably, be quite interested in a test on which the majority of our 
students had nearly perfect scores-assuming that our efforts at in- 
struction were successful. In fact, long before I took measurement 
courses there used to be a term for such instruments. I believe they 
were called "mastery" tests. 

But the matter of test difficulty is only one problem associated 
with the use of achievement tests in evaluating instructional vari- 
ables. The bald fact o£ the matter is that scores on achievement 
tests are just generally insensitive to particular educational experi- 
ences. Achievement tests tend to reflect rather siaUe characteristics 
of the learner. This is one of the reasons such tests are reliable, and 
it constitutes an absolutely essential condition for the evaluation of 
individuals. Nevertheless, there appears to be a rather pervasive 
belief that intelligence tests and achievement tests are very differ- 
ent sorts of measures, that achievement tests reflect school learning, 
while intelligence tests do not. Actually both types of" instruments 
reflect school learning, but mainly in the sense of being related to 
more general variables such as number of years of schooling in the 
case of both types of measures, or whether the student has taken 
a given subject in the case of achievement tests. After all, the high 
correlations bet\veen aptitude tests and achievement tests reported 
in test manuals and many research studies ought to convince us that 
the two types of tests rdBect many of the same aspects of students, 
both cognitive and motivational. All of this adds up to the fact that 
the use of total scores on achievement test in the evaluation of in- 
stnictional programs often may insure negative results, even when 
the program might be operating quite effectively. 

You might have the idea by now that I am saying that achieve- 
ment tests should not be used in evaluation studies. This is not 
true at all. I think that standardized achievement tests ought to have 
extensive application to the evaluation of instructional programs. 
What I am objecting to is the uncritical use of total scores on 
achievement tests in the belief that such scores will be sufficiently 
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sensitive to what goes on in the classroom to provide a basis for 
deciding on the effectiveness of particular educational practices. 
If we forget about total scores, however, and look at the level of 
the individual test item, it is quite possible that the achievement 
test will turn out to be a very useful tool in evaluation. Why? Well, 
although total scores on a test or scale may be little affected by the 
program we are studying, a few of the items may show dramatic 
changes in proportion passing or in correlation with some relevant 
criterion. Presumably, these items would be more closely related 
content-wise to the program under study than is the test as a whole. 
At the purely empirical level, then, analysis of individual items 
would provide us with' a means of discovering what particular 
learning has occurred. Such specific information is far more mean- 
ingful and relevant to the task of evaluation as defined here than 
in any total score that I know of. If we combine this search for 
changes on individual items with what are now referred to as 
"item-sampling" procedures, the latter allowing us to administer 
very large numbere of items, but each item to only a small num- 
ber of individuals, we then would have an evaluation tool of really 
interesting potential. I would admit that the tool looks suspiciously 
like a shotgun, but at least it is a vastly bigger and more powerful 
shotgun. 

W^ile I may have been somewhat hard on the researcher up to 
this point, I think we all recognize that there is another side to the 
coin. The hardheaded research type, if we can induce him to involve 
himself in the evaluation of what we do in our schools, may have 
some points of view, in addition to his technical competence, that 
will make both a unique and significant contribution. I would like 
to speak about two charact-^ristics which the researcher, perhaps 
more than anyone else, has to offer. 

First of all, the researcher who is worth his salt is likely to be 
interested in at least some aspects of theory in the behavioral sci- 
ences. If such interests influence his approach to the evaluation of 
instruction, and they should, the researcher is likely to pose some 
fundamental questions about what the program is accomplishing- 
questions that otherwise might not have been asked. For example, 
the developmental psychologist may be interested in finding out 
whether a given program can teach first graders to perform certain 
arithmetical operations-straightforward instructional goals-but he 
may also become curious as to whether the instruction has any 
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influence on cognitive processes at a level more basic than instruc- 
tional content. For example, Piaget, whether or not you accept his 
theories about cognitive growth, sees the development of certain 
conceptions as naturally antecedent to learning the formalized oper- 
ations of arithmetic as well as other rather different, educational 
content. One v/ould expect that a researcher interested in these 
more fundamental concepts and processes would manage to find 
ways to study them as a part of his efforts at evaluation. On the 
one hand, he might collect evidence confirming Piaget's notions 
that children who have not developed certain fundamental skills 
cannot learn arithmetic. This information would be useful to the 
developer, and also would make a contribution to knowledge in 
what is usually referred to as basic research. On the other hand, the 
researcher might discover that the program influences the develop, 
ment of the fundamental cognitive processes he is studying. Such 
a finding would obviously be of even more importance to all con- 
cerned. 

The point of all this is that research, antj research, involves asking 
questions. The researcher is a man who, if he is any good, asks the 
most interesting questions— even some questions that, at first sight, 
might not appear to be relevant to the practical problem of making 
a program work. At best, he will relate the evaluation of instruc- 
tional procedures or techniques to issues that transcend immediate 
practical concerns. 

A second contribution that is in the special province of the re- 
searcher — and again this is in addition to technical know-how — also 
involves the questions asked in evaluation. Scriven^ (1965) has 
pointed out very forcefully that there is a widespread tendency to 
'Velativize" the evaluation function— that is, there is often pressure 
to restrict evaluative information to criteria generated primarily by 
the particular educational program under study, rather than by in- 
structional goals with broad generality. This, to be sure, is an im- 
pressionistic observation. It cannot be verified by citing a U. S. 
Office survey of the goals of evaluation projects over the past five 
years. But I think that anyone who has had much contact with 
developers wfll have run into this tendency to view many criteria as 
either biased or irrelevant to the program under study. 

Developers are naturally enthusiastic about their own ideas and 
son^-^Hmes rather scornful about what has been done in the past. It 
is especially significant that not only are earlier instructional methods 
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viewed as inadequate, but earlier goals as well It is often possible 
that the developer's enthusiasms will limit the thrust and scope of 
the questions posed in the evaluation. 

I view this tendency to "relativize" evaluation goals as, first of all, 
unrealistic from tlie pragmatic point of view. It is obvious that 
concentration on a limited set of instructional goals in evaluation is 
likely to limit seriously the spin-ofiE of discoveries in the behavioral 
sciences. But sticking with pragmatics, we ought to recognize that 
people making decisions about adoption of the program in schools 
may have questions about other goals. We may accept the instruc- 
tional intent of a modem mathematics curriculum, but still wonder 
how students taking the new curriculum fare in terms of the goals 
of "traditional" mathematics instruction, whatever they may be. To 
agree that the traditional criterion may be biased against students 
who have taken the new curriculum is emphatically not a reason for 
failing to include that criterion in our evaluation. The presumption 
of bias requires us to be careful and responsible in our interpreta- 
tions, but does not indicate that the information provided is without 
meaning. 

It is my belief that concentration on a program-specific set of goals 
will run against the grain of the average educational researcher. 
If he is at all theoretically oriented, he should be accustomed to 
thinking in terms that are considerably broader than particular in- 
structional content. Thus, the most important strength the researcher 
brings to his role in evaluation is not his knowledge of psycho- 
metrics, or statistics, or experimental design, but the simple fact 
that he is by nature a creature who asks questions. In the light of 
this, it would be helpful if diplomacy were also one of his strengths, 
because those questions may, at times, make the developer uneasy. 

We have seen tliat the behavioral scientist may have some things 
to learn before he can perform eflFectively in his role in the evalua- 
tion of instructional programs, it is equally true that he has signif- 
icant tilings to offer in addition to his technical competence. But 
one cannot help but wonder how the evaluation researcher's findings 
contribute to final decisions about which innovations are to be re- 
tained in the schools. It may be that this sudden escalation in our 
evaluation requirement is creating a new role for the educational 
researcher, one that is somewhat closer to the final decision process 
than any has played previously. 

Now, the school is, and always will be, a socio-political institu- 
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tion. The influence of the researcher on the men and women who 
make decisions about curriculum, school organization, and the like, 
is, in one seme, similar to the economist's influence on the govern- 
ment official. A tax boost this year may be excellent economics, but 
nonsensical politics and, thus, overruled by that official. I am sure 
that the same situation often pertains to many of the innovations 
we in educational research see as imperatives. Yet, in another sense, 
I do not feel that educational researchers in the past have had quite 
as formalized a role in the educational decision-making process as 
economists and other experf^s have had in government. This is not 
to say that research has not had a genuine impact on educational 
policy. It most emphatically has. But the influence has perhaps more 
often conformed to the traditional '^pipeline" model in which ideas 
or inventions are generated in the scholars armchair or laboratory, 
go through a developmental phase (usually in the hands of others), 
and then, as formalized procedures, go through dissemination and 
adoption phases. Insofar as this model is appropriate, it portrays a 
rather wide separation between the scholarly researcher and the 
educational decision maker. 

Much of the substance of what I have had to say this afternoon 
amounts to a contention that the evaluation researcher, if he is to 
be eflFective, will have to get a lot closer to the school in all its 
aspects than he has in the past. It seems to me that this reqtnrement 
is also going to put many researchers very much closer to the times 
and places where decisions are made. In other words, the researcher, 
perhaps in spite of himself, has been washed a considerable dis- 
tance closer to the other end of the theoretical pipeline. 

I have a sociologist friend, who, the other day, explained to me 
that all of these obser\'ations are perfectly obvious to anyone who 
is fortunate enou^ to be a sociologist. If I understood him cor- 
rectly, and there is some question as to whether I did, it works 
something like this. Roles are defined by people's expectations. In 
other words, there is some sor*^ consensus that people who occupy 
a given role ought to behave in certain ways. Much of this paper 
deals with what we have a right to expect from the evaluator. Al- 
though defined in terms of expectations, roles are created to fulfill 
functions, that is, to accomplish something. Functions, in turn, stem 
from demands made by society. You may say, "Well, what demands 
has American society made that have generated or at least stimu- 
lated tlie evaluation role?" I think the answer is clear. Very powerful 



ARE EDUCATIONAL RESEARCHERS PREPARED? 47 



voices in our society are saying, "Do something to adapt the schools 
to the educationally deprived!" "Do something about incorporating 
the new technology into instruction" and "Do something about ap- 
plying new ideas about the learning process." Political leaders have, 
in part, responded to these demands with the Federal Education 
Act of 1965. With its provisions for the annual evaluation of educa- 
tional programs that are already in operation this Act has gone a 
long way toward creating a new role in education for the behavioral 
scientists, one that is clearly closer to the places where decisions 
are made. If we who are in research continue to take some time to 
think about the nature of this new role, we will be much better 
equipped to fulfill it. 
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Responsibility in Innovation 



LELAND B. NE\VCOMER 

I feel I am in a rather ridiculous situation. I am not a scholar and 
I feel just a little bit out of place. I guess if I were to attempt to 
classify myself I would like to be known as a "manager of educa- 
tional enterprises" or maybe an "organizer of time, space, people 
and things'" or perhaps a "politician." 

There are a lot of people who are superintendents who would not 
admit that they are politicians and I feel that that is one of our 
problems in education today. We have superintendents who are not 
too honest-dont misunderstand me-by that I don't really mean 
a moral virtue, I mean they tend not to face up to the critical issues 
of our time and they don't face up to the fact that public education 
is in the political marketplace. We educators are in the heat ui the 
kitchen. I for one think' this is good. We have been fighting f x 
many, many years to get there and have been saying how impor- 
tant education is. It is the most important thing that goes on. 

If you include what is called the Toiowledge industry"-book 
publishers, publishers of instructional materials, technology of var- 
ious kinds, including the expense of public education, you have the 
biggest industry in the amount of dollars spent in the whole of the 
United States, even surpassing the defense effort, the war effort 
and everything else. 

With this being tnie, we are kidding ourselves if we dont face 
up to the fact that we are going to have to be held accountable. 
We arc in a position where somebody is going to be looking at us 
all of the time. If we don't choose to be in that kind of dilemma I 
suspect we should not choose to be in public education or at least 
not choose to be a superintendent. 
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I really happy to have the opportunity to talk here. I relish 
the opportunity, not because I feel qualified, but because I do have 
a lot of ideas which, although Tm sure I can't prove and w^hicl. are 
not even statistically measurable, are ideas that people ought to 
tliink about. I am not even convinced about them myself but I talk 
about them as though I am very much convinced because that tends 
to make people react more violently and stirs up their adrenalin a 
little bit so that more thinking goes on. 

I am one of those people just crashing for innovation. I w^ant to 
innovate all over the place. I think it is essential. I think the v/ay 
teaching and learning goes on nov/ is pretty obsolete, really. I got 
into trouble the other day by saying that we w^ere operating at 50 
percent efficiency so I wont say that today. But the important thing 
I want to talk about is "Responsibility in Innovation." I am not 
going to talk about sophisticated research or sophisticated evalu- 
ation because I really don*t knov^^ anything about that. I am going 
to attempt to talk about the organization of schools. I don t knov^^ 
much about things other than schools. I would like to talk about 
an organization of schools which will increase the amount of, and 
the spoed of, change or innovation. I hope that this organization 
will increase the chances that what we do differently will make a 
difference— a difference toward achieving the objecti /es we have. 

TTie other day I was talking to Dr., Harold B. Gores, president of 
the Educational Facilities Laboratory, a Ford Foundation acvivity. 
Dr. Gores is an extremely popular man. His job is to give away mai 
millions of dollars each year to people who can come up with rig- 
nificant ideas which might tend to increase the chances of making 
more efficient use of the facilities in our school buildings. He said, 
"You know, wc are in a fantastic revolution. More things are going 
on now — things are 'popping out* all over. There is tremendous 
pressuiv. There is a fantastic increase in the amount of funds pro- 
vided by the Federal Government and this creates many problems. 
It creates excitement, it creates resource^ and it creates action.** 

I like action. Imagine Title I, II, III, IV, V, ESEA and all the rest. 
Imagine knowing that all that money is there and imagine the 
problems being created. What do you supp goes through people's 
minds? I have a sneaking suspicion that what tends, on too many 
occasions to go through the local districts' mind is, "How can we 
get some of that money?** rather than looking at the situation and 
saying, "What problems do we have for which a sound proposal 
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can be made/' "How can we define the problem in sucli a way that 
we can find the best source for its funding?" 

Now, if we do tins then I suspect that -.vhat we are doing is going 
to really enhance and increase the effectiveness of teaching and 
learning. If we take the other route "Get the money" then we are 
going to end up in just doing one thing and that is, proving how 
incompetent we really are. We have been screaming for money for 
a long time. Now we are getting money and if we don't produce 
something with that money it will be the coup de grace for edu- 
cators as managers of schools. 

You have read Ralph Hemling's article "Is the School Superin- 
tendent Obsolete?" He builds a pretty good case. I don't agiee with 
him, of course, but irresponsible innovation is the best possible way 
that I know of to make the educational enterprise or establishment 
as it now exists, obsolete. 

Let me get more into what I want to talk about. A dilemma exists. 
We have people running in two directions. We have those that are 
saying "be careful"-these are the reactionary ones. They are think- 
ing of 125 reasons why they shouldn't or can't be doing something. 
Then we have those people that say, "Don't just stand there. Do 
something! Do something!" You know, those are the ones that say 
"Are you doing team teaching?" "Do you have ungraded class- 
rooms?" "You don't?" "Isn't that too bad." "Do you have continuous 
progress in your schools?" "Do you have modular scheduling?" "Do 
you have flexible scheduling?' "Do you have independent study 
time?" "Do you have . . .?" We could go on and on and on. They are 
becoming cliches. We have people who are just busting out inno- 
vating all over the place because they think it is the "George" thing 
to do and then we have those who are dragging anchor. 

Why does this take place? I say it is because of poor management. 
You see, T am prejudiced. I think good superintendents are essential 
if we are going to get something done, but their primary task is not 
to do things. It is to organize an environment in which these things 
will happen and happen effectively. 

I am sure I don't know exactly how to do this but I have some 
ideas that I want to discuss with yvou. Before I get into that spe- 
cificall) though, people have talked about the technological revo- 
lution and the knowledge explosion and I think they have been 
referred to today so I don't have to dwell on that. Just to make this 
one point, I really believe tliat we are kidding ourselves if we think 
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the present organization and the present way we approach public 
education will allow us to continue effectively. As we are presently 
organized it is impossible to effectively take advantage of the 
knowledge and the hardware that we already have in our schools. 

'Hie reason I say this is because as I look at school districts I have 
to assume by the way they are organized, that they are organized 
for the purpose of control. I have never seen an organization that is 
so well set up to be conservative or reactionary. Local boards of 
trustees are reactionary. Local boards of trustees have a tendency 
to think that when they get five phone calls or when 20 people come 
to a board meeting and express a point of view that this feedback 
represents the universe. You get action that way, you know. Well, 
this kiiid of thing creates a very real problem because most of our 
schools are organized this way. 

I would suggest to you that perhaps schools should be organized 
at hast 50/50--that is 50 percent of the resources going to control 
(keeping the store, operating the establishment) and 50 percent of 
the effort or power or prestige or whatever you want to call it, going 
to '*break-through— going to the problem of improving the opera- 
tion. Now I again submit that if this doesn t happen we will become 
obsolete. Somebody elsj is going to take over the function of edu- 
cation in our society. J firmly believe this although I don't approve. 

The real revolution, as I see it, aside from this organization prob- 
lem, is a revolution away from teaching to learning. That is an over 
simplification and I am sure some of you people can poke holes in it. 
I hope you are thinking about it right now. But the emphasis used 
to be on teaching. When I first started observing classrooms and 
doing, so called, curriculum supervision, we would go into a class- 
room and would evaluate or manage by process or by method. I 
call it "characteristic evaluation." We would note, "Does the teacher 
smile?" If the teacher smiles a lot then we infer that the teacher must 
be warm. If the teacher is warm then that ought to be good for 
kids and so we say that probably if she smiles it increases the 
chances that kids learn. 

Then, if there are centers of interest, if the pictures are straight 
and the room environment is good— boy, we really worked on room 
environment—we thought it was good. Dont get me wrong. All 
these things have some kind of validity but unfortunately that wai, 
the only thing we were using. We were really looking at process and 
method all the way. 
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What is the net result of an enterprise that evaluates or manages 
by the methods used? What happens is that it evolves toward man- 
agement by prescriptioa That means that you tell teachers what to 
do and how. The one thing I fear today is that most of the innova- 
tions we have going on now are becoming prescription innovations- 
team teaching, modular scheduling, flexible scheduling— all these. 
Somebody who knows all is prescribing a new means and then the 
people who are to implement the means are becoming like the 
people that implemented the means before. We attract people who 
just want to follow the leader.. 

I think John Dewey was great. I think he had some of the greatest 
ideas that have ever been contributed to education. But some of his 
disciples, in my opinion, really fouled up these ideas. 

A funny thing happened when I began teaching The rage then 
was the social studies tmit of work. This was an innovation and it 
was creative and I think it was sound, but you know, before we got 
through it was more traditional and more crystallized and more 
rigid in the prescription of method than anything I have ever seen. 
It started out as an objective but it ended up as a prescription of 
method. This is the thing I am worried about. And I think the revo- 
lution from teaching to learning is when we put our attention to 
learning then we can stop thinking so much about the means as we 
develop the ability to measure the results of learning. 

We haven't gotten there yet. Here I am getting way out of my 
field. I am getting into your field so I am on dangerous ground. 
I don t think we can measure very many things, but I think there is 
an in-between step between evaluation or management by method, 
prescription or process and evaluating or managing by results. And 
that I call "managing by objective/' Earlier Mr. Skager mentioned 
that the true researcher should ask the right questions. Now, if you 
can organize so that questions are asked in such a way that the 
objectives become really clear and you can manage by objective, 
I think you have made a very strong step toward the eventual man- 
agement by results. 

Let me give you a kind of facetious illustration of what I mean in 
connection with this. I have an objective. I have a problem. My 
problem is that I hate Board meetings. I hate Board meetings with 
a passion. I should probably never have been a superintendent be- 
cause that is a dilemma for me. Having Board meetings is one of 
the functions which superintendents have to be involved in. But I 
hate them and that constitutes a problem for me. 
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How about a plan or an objective— first an objective. An objective 
might be to somehow, instead of having Board meetings about every 
week or two (with a special one in between) to have them once 
every three months or even every six months as the big corporations, 
such as General Electric do. That would really be something. So that 
becomes an objective. 

Now a plan. You look at the corporation and say, "How can they 
do that?" I will tell you how they can do that. In the first place they 
can do it because they go in every six months or every year for their 
annual meeting or whatever it is and present an evaluation of the 
performance of the organization in terms of i profit and loss state- 
ment. If they make a profit, great. That is really their objective. 

We in school have a problem. We aren't a profit making operation 
and we haven't learned how to evaluate our product in terms th^^t 
the community will accept as evidence that we are achieving the 
objectives they desire. Now, we should be able to do that. I have a 
sneaking suspicion that I could get Board meetings down to once 
every three months or once every six months because I could say 
to them, '*Now, look. Let s face it. This we have clearly defined is 
what we are here for. We agree. O.K.? O.K." We go the next step. 
We say, "Now, this is what you will accept as evidence that we are 
doing what you think we should do-tifiat the product is as you 
think it should be? This is the evidence." Then, I produce the 
evidence. 

And so, every three months we look at it and we see whether it 
is going in the right direction, whether it is good or bad. If it is bad 
the Board says, "Buddy, you have three months to get it started the 
other way.'* If it is good you might ask for a raise. 

Every three months, then the Board members and the community 
will get their minds oflF tifie prescription method. They won't be too 
concerned about what methods are being used as long as kids are 
Ir^aming effectively. 

This is why I am so excited about the N?itional Assessment Pro- 
gram. I am for it and what they are trying to do. It is going to open 
a can of worms all right and we are going to have a few problems 
but I have learned one thing since I have been trying to be a man- 
ager and that is that one of the best ways to get something accom- 
plished is to create a problem. You know, create a big fat problem. 
In fact I told the Board members at Newport-Mesa, **If you want 
somebody to come in and run a smooth ship, don t hire me because 
I plan to create more problems than I solve." I think this is one of 
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the functions of leadership. I think that national assessment is going 
to create some problems but for heaven's sake, what kind of an 
assessment do we have now? Twelve people come to a Board meet- 
ing and the Board thinks that they are the universe. and so we 
change. Is that the kind of input we want to determine, the direction 
and die gcals of our schools? 

Lets get back to the specific organization. I think that the large 
central oflSce, the big research department, the grandiose assessment, 
the large innovations, the large program change or improvemenl, 
district-wide operations are passe. You know, it used to be that you 
could set up a committee in a large school district and then spend 
a year defining the objectives and then four years later you would 
implement a new reading program, a new literature program, a new 
this-program or that-program and then every four or five years, you 
would take on c different one. That's ridiculous. Text books are 
changing faster fJian that. The innovation has to be at the grass- 
roots. How to get re.sponsible innovation at the grassroots of an 
operation becomes the real question. 

I will tell you how I think it ought to be done. Let's just take a 
typical organization of the school district. I think if a school district 
is large enough to have an assistant superintendent they ought to 
have two. One of them ought to be in charge of operations, an assist- 
ant superintendent in charge of operations. He is like the executive 
oflBcer aboard a ship. He runs the everyday operations. He leaves 
the captain of the ship free to plan and determine the course and 
the variety of other kinds of things. 

The other one would be the assistant superintendent in charge 
of "break-through,** but let s be a little more traditional and call him 
''research and development." But "break-through" is why he is there. 
He ought to be right alongside the other assistant superintendent 
and he ought to be paid just as much and he ought to be just as 
mudh in the line, not staked out in a staff operation some place, but 
right there in the main stream. His function is to foster break- 
through and responsible innovation. 

Now let's take it out of the district office. Let's go out into the 
high school. By the way, I could talk the whole time on the high 
school if you would like. I think that is the most beautiful example 
of Parkinson's Law I have ever seen outside of PTA. 

Look at the organization of the high school and the way £hey 
have added positions to take care of things that happen without 
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ever looking back. Let me give you an example. The high schools 
where I am right now have a principal, an assistant principal and 
two vice principals. They have only had an assistant principal for 
about four years. Do you know how they got that assistant principal? 
They decided they were going to do something to upgrade teaching. 
They wanted to pay good teachers more than poor teachers and they 
weren't auie to let poor teachers go so they had a problem working 
this out. They said, ''We have to have another staflF member," so 
they hired an a^^sistant principal. But you know, as it worked out 
they didn't establish their criteria, they didn't know how to evalu- 
ate and determine which was wliich and the program failed. But 
do you know what? T\ie program failed two years ago and they 
still have the assistant principal. That's Parkinson s Law in operation. 

But back to research and development* You ought to have re- 
search and development right at the high school. You have a prin- 
cipal and you may well have a very high-level p**rson who is the 
executive officer or whatever you want to call him. The assistant 
principal or vice-principal in charge of operations runs the store and 
keeps it going well. Then you should have the other one, the assist- 
ant principal or vice-principal or whatever you want to call him, in 
charge of break-through, research and development, right there on 
that campus-the research person. There aren't enough of them I 
grant. We probably aren't training enough to do this all the way but 
I bet we are training enough who know how to play the Devil's 
Advocate and who, if we put them in the right spot in the organi- 
zation with the right kind of power, could ask the uncomfortable 
question. Why do you do it that way? Because the best approach 
to responsible innovation is in having somebody play the Devil's 
Advocate. 

In the traditional management sense this is called ''decentraliza- 
tion.'* I believe that if we are going to have responsible change in 
typical schools, not in just the three or four would-be isolated schools 
tiu-oughout the United States, but responsible change going on 
every place at a pace fast enough to keep us from losing whatever 
it is we are working for, we are going to have to buUd fast. To do 
this we will have to take advantage of what I think is the most 
precious thing in the world, and that is the human being— his brain 
and his intelligence. We are getting better teachers all the time. 
Fountainheads must come up all over the place. We can no longer 
prescribe. Teachers are getting militant. Do you know why they 
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are getting militant? Because they are being treated paternally, like 
peons. They really are. It isn't really money they want so much as to 
be treated as professionals. It is touch and go as to whether or not 
they are going to be a professional or whether they are going to go 
another route. It is touch and go as to who is going to determine 
which way it is gping to go. Our ability as administrators and staff 
people and psychologists is to help and to learn how to work with 
teachers to bring about positive change, right there at the grass- 
roots level of the operation. I feel so strongly about this that I guess 
I sound like a Baptist minister when I talk about it. 

Let me get to one other idea. We are trying to work out some- 
thing that, for want of a better word we are calling it a ^'develop- 
ment laboratory." It's a kind of a phenomenon in a school district. 
It's a place where, when somebody has an idea, somebody will be 
there to help them to put it into some kind of a plan or format to 
bring it to fruition. It's a place not to generate ideas and impose 
them on other people but to help teachers give birth to their own 
ideas. Too often a teacher who has an idea goes to his principal and 
says, "I have an idea I want to try' out. I think it would be terrific. 
I think it would improve the efficiency of our operation." Do you 
know what they are asked? "Does it cost any money?" ''Well, not 
much, but just a little bit." "Is it in the budget?" "No. No. I guess 
it isn't in the budget." ''Well, then, I will tell you what you do. You 
just keep it hot and next spring we wUl see whether it can survive, 
when we start — Now, you know how hot it will be. Teachers are 
busy people and that idea will get cold ver; fast. 

What would happen, now, if right in your organiz ition, in your 
budget, not as an adjunct, but right in the middle of it, you put the 
development lab where somebody can go when he gets a hot idea? 
Now, I don't want to make it easy. Make them stand the test, have 
a panel of interrogators or something, or a Devil's Advocate, poking 
holes and trying to show that the idea isn't any good. But, then, if it 
does have some merit, fund it with your 'Tiot idea" fund. 

Some people think that is ridiculous. However, it will work. We 
call it the 'liot idea fund." Fifty thousand bucks. You know, people 
learn that the fund is there, but, then we have a problem. Do you 
know what? Teachers and principal don't know how— I shouldn't 
say this. I am not being negative about it when I say this. They are 
busy people. They are not supposed to know how to organize 
projects, but they can be helped. When you dangle $50,000 out 
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there, it's the dog-gonedest motivation. You know, its a lot better 
motivation than saying, ''Why don't you do something different in 
your classrooms?*' 

You dangle this and they want some of it Then you set the con- 
ditions. You have the right to. The money is there and you have 
every right to set some conditions. The conditions are that they 
must have a plan and the plan must include objectives. The plan 
must include a plan to accomplish the objectives. And then it must 
include something else. "How are you going to kiiow when you get 
through that it made a difference?" Now, those are simple words. 
Those aren't even statistical words are they? But the teachers under- 
stand those words and they understand them real well. They get 
excited about it. 

I think my whole thesis-and I am practically through-is that 
you can organize to improve. It depends upon what you value and 
where you place that value. You can organize so that teachers and 
principals will work on res{>onsible innovation if you dangle the 
carrots in the right places and if you establish a system to get it done. 

Let me mention one more thing. It is a little rough to have a full 
time research and development plan with an assistant in every 
school at the elementary level. I believe in establishing the position 
of a "learning analyst** in each school. If our function primarily is 
learning and we provide for a principal and teachers, then this is 
somebody whose job is to help them analyze whether they are 
achieving their goals or not. Then you evaluate the principal and 
the teachers in terms of whether they achieve those goals. I think 
in this way you have some action going. You establish a climate for 
human dialogue, not cold statistics. You have somebody asking 
questions. A teacher recently reported about how excited the first 
grade teachers were because somebody asked and talked about what 
they were doing. They need this and they need somebody to put it 
in the kind of terms that will allow Uiem to know when they get 
through whether it made any difference. 

Now let me close by saying that, and I say this because I feel it 
strongly, with all the innovation and all of the things that we work 
on to improve education, there is just one thing we must never 
forget— the most important thing that goes on in any school occurs 
when a kid gets to feel good about himself. Whenever this happens 
(and I think you know what I mean) we have achieved. So, along 
with all of this you must continually work on making absolutely 
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sure that every kid that goes to school finds somebody who cares. 
This is more important, I think, than any organization of time or 
space or resources or anything else. And the innovation that we work 
on is really an attempt to try to work out a better way to accomplish 
this particular goal. 

I wish we would quit talking about process all of the tinie. I will 
close with one little illustration. I am putting a lot of pressure now 
on trying to get some things going to increase efficiency, particularly 
at the high school level. A principal came to me the other day and 
said, "Well, I'm ready." And I asked, 'Tou are ready for what?" 
And he said, "I am ready to go into flexible scheduling next fall.** 
And he really expected me to say, **My God, man, that's great. You 
are really it." I said, **VVhat in the heck do you want to do that for?** 
He was frustrated. He said, **What do you mean? You have been 
screaming around here about innovating and getting things done.** 
And I said, "Well, now, what in the world would you want to go 
into flexible scheduling for?** He really didn't have a very good 
answer. I am sure that the real reason he wanted to go into flexible 
scheduling was because he thought I wanted him to. I said, "It's the 
last thing I want you to do. Buddy. Let's not go on flexible schedul- 
ing there unless it helps to meet our primary objective— learning." 
He walked out talking to himself. 

It's going to take some time to get to this point where improving 
school learning is the target. But I repeat what I said originally— 
you need to create some problems in order to come along with some 
good solutions. 
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